Smart controlling device and method of controlling therefor

ABSTRACT

The present specification relates to a smart controlling device capable of utilizing machine learning for voice recognition and a method of controlling therefor. The smart controlling device according to the present invention includes a receiver configured to receive an input including a command trigger, and a controller configured to detect one or more external display devices, select a display device of the detected one or more external display devices, cause a power status of the selected display device to be changed to a first state, and cause a response data corresponding to a first command data received after the command trigger to be output on a display of the selected display device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/815,491, filed on Nov. 16, 2017, now U.S Pat. No. 10,489,111, whichclaims the benefit of earlier filing date and right of priority toKorean Patent Application No. 10-2017-0039299, filed on Mar. 28, 2017,the contents of which are all hereby incorporated by reference hereintheir entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an audio device for processing dataaccording to a voice recognized using machine learning and a method ofcontrolling therefor.

Discussion of the Related Art

Generally, terminals can be classified into mobile terminals andstationary terminals according to a presence or non-presence ofmobility. And, the mobile terminals may be further classified intohandheld terminals and vehicle mount terminals according to the abilityto hand-carry a device.

Recently, functions of a mobile terminal tend to be diversified. Forinstance, the diversified functions may include data and audiocommunication, photographing and recording a video through a camera,audio recording, music playback through a speaker system, outputtingimage or video through a display unit, and the like. A prescribedterminal may be further equipped with an electronic game play functionor perform a multimedia player function. Particularly, mobile terminalsmay also be able to receive multicast signals for providing visualcontent such as a broadcast, a video, a television program and the like.

As functions of the terminal are becoming increasingly diversified, theterminal tends to be implemented as a multimedia player provided withcomplex functions such as photographing of photos or videos, playback ofmusic or video files, game play, broadcast reception and the like forexample.

To support and increase the terminal functions, improvement ofstructural parts and/or software parts of the terminal are considered.

Recently, audio devices capable of performing voice recognition havebeen developed. The audio device may be equipped with a speaker systemand be able recognize voice inputs and perform a voice-relatedoperation. The audio system may also control home appliances bycommunicating with the home appliances capable of communicating with theaudio system. Hence, a user can conveniently execute various functionsof the home appliances via voice input to the audio device.

However, since the audio device operates in accordance with an input andan output of audio data only, there is a problem of sufficiently copingwith various needs or intentions of a user.

SUMMARY OF THE INVENTION

Accordingly, the present invention is directed to an apparatus andmethod that addresses one or more problems due to limitations anddisadvantages of the related art. An object of the present invention isto provide a smart controlling device capable of recognizing an inputsignal, analyzing the recognized input signal, and performing anoperation corresponding to the input signal.

Another object of the present invention is to provide a smartcontrolling device capable of easily and quickly processing an inputsignal, although the input signal is not restricted to voice and anoutput signal is not an audio signal.

The other object of the present invention is to provide a smartcontrolling device capable of satisfying an intention of a speaker(user) and enhancing convenience by providing the speaker with anadaptive response based on an event such as a status of the speaker, andthe like using a display of a different device adjacent to the smartcontrolling device.

Technical tasks obtainable from the present invention are non-limitedthe above mentioned technical tasks. And, other unmentioned technicaltasks can be clearly understood from the following description by thosehaving ordinary skill in the technical field to which the presentinvention pertains.

A smart controlling device capable of performing voice recognition and amethod of controlling therefor are disclosed in the presentspecification.

To achieve these objects and other advantages and in accordance with thepurpose of the invention, as embodied and broadly described herein,according to one embodiment, a smart controlling device includes areceiver configured to receive an input comprising a command trigger,and a controller configured to detect one or more external displaydevices; select a display device of the detected one or more externaldisplay devices; cause a power status of the selected display device tobe changed to a first state; and cause a response data corresponding toa first command data received after the command trigger to be output ona display of the selected display device. To further achieve theseobjects and other advantages and in accordance with the purpose of theinvention, as embodied and broadly described herein, according to adifferent embodiment, a method of controlling a smart controlling deviceincludes receiving an input comprising a command trigger; detecting oneor more external display devices; selecting a display device of thedetected one or more external display devices; changing a power statusof the selected display device to a first state; and outputting responsedata corresponding to a first command data received after the commandtrigger on a display of the selected display device.

A computer-readable medium for controlling a smart controlling device isprovided. The computer-readable medium may include computer executablecode. The computer-readable medium may include code for receiving aninput comprising a command trigger; detecting a location of a source ofthe received input; detecting one or more external display devices basedon the detected location; selecting a display device from the detectedone or more external display devices; transmitting a control command tochange a power status of the selected display device to a stand-bystatus; generating response data corresponding to command data receivedafter the command trigger; and transmitting a control command forswitching a power status of the selected display device to a power-onstatus and displaying the generated response data via a display of theselected display device when it is determined that the generatedresponse data is required to be displayed on a display.

Technical solutions obtainable from the present invention arenon-limited the above mentioned technical solutions. And, otherunmentioned technical solutions can be clearly understood from thefollowing description by those having ordinary skill in the technicalfield to which the present invention pertains.

According to at least one of various embodiments of the presentdisclosure, it is able to cope with an input signal by recognizing theinput signal and analyzing the recognized input signal.

According to at least one of various embodiments of the presentdisclosure, it is able to easily and quickly process an input signal,although the input signal is not restricted to voice and an outputsignal is not an audio signal.

According to at least one of various embodiments of the presentdisclosure, it is able to satisfy an intention of a speaker (user) andenhance convenience by providing the speaker with an adaptive responsebased on an event such as a status of the speaker, and the like using adisplay of a different adjacent device.

Effects obtainable from the present disclosure may be non-limited by theabove mentioned effect. And, other unmentioned effects can be clearlyunderstood from the following description by those having ordinary skillin the technical field to which the present invention pertains.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the invention and are incorporated in and constitute apart of this application, illustrate embodiment(s) of the invention andtogether with the description serve to explain the principle of theinvention.

FIG. 1 is a schematic diagram for a digital system including a smartcontrolling device according to an embodiment of the present disclosure;

FIG. 2 is a block diagram for a smart controlling device according toone embodiment of the present disclosure;

FIG. 3 is a diagram for explaining an exterior of a smart controllingdevice according to one embodiment of the present disclosure;

FIG. 4A is a block diagram for a configuration of a digital TV accordingto one embodiment of the present disclosure;

FIG. 4B is a block diagram for a configuration of a digital TV accordingto a different embodiment of the present disclosure;

FIG. 5 is a block diagram for a configuration of a mobile terminalaccording to one embodiment of the present disclosure;

FIG. 6A is a conceptual diagram for an execution state of a voicerecognition function;

FIG. 6B is a flowchart for a method of executing a voice recognitionfunction;

FIG. 7 is a flowchart for explaining a method of processing data of asmart controlling device according to an embodiment of the presentdisclosure;

FIG. 8 is a flowchart for explaining a method of processing data of asmart controlling device according to one embodiment of the presentdisclosure;

FIG. 9 is a diagram for explaining a method of controlling a displaydevice of a smart controlling device according to one embodiment of thepresent disclosure;

FIG. 10 is a flowchart for explaining a method of outputting data of asmart controlling device according to a different embodiment of thepresent disclosure;

FIGS. 11, 12, 13, and 14 are diagrams for explaining various scenariosof using a smart controlling device according to the present disclosure;

FIG. 15 is a flowchart for explaining a method of outputting data of asmart controlling device according to a further different embodiment ofthe present disclosure;

FIG. 16 is a diagram for explaining a scenario including imageinformation of a smart controlling device according to the presentdisclosure;

FIG. 17 is a flowchart for explaining a method of outputting data of asmart controlling device according to a further different embodiment ofthe present disclosure;

FIG. 18 is a diagram for explaining a scenario including imageinformation of a smart controlling device according to the presentdisclosure.

DETAILED DESCRIPTION OF THE INVENTION

Description will now be given in detail according to exemplaryembodiments disclosed herein, with reference to the accompanyingdrawings. For the sake of brief description with reference to thedrawings, the same or equivalent components may be provided with thesame reference numbers, and description thereof will not be repeated. Ingeneral, a suffix such as “module” and “unit” may be used to refer toelements or components. Use of such a suffix herein is merely intendedto facilitate description of the specification, and the suffix itself isnot intended to give any special meaning or function. In the presentdisclosure, that which is well-known to one of ordinary skill in therelevant art has generally been omitted for the sake of brevity. Theaccompanying drawings are used to help easily understand varioustechnical features and it should be understood that the embodimentspresented herein are not limited by the accompanying drawings. As such,the present disclosure should be construed to extend to any alterations,equivalents and substitutes in addition to those which are particularlyset out in the accompanying drawings.

It will be understood that although the terms first, second, etc. may beused herein to describe various elements, these elements should not belimited by these terms. These terms are generally only used todistinguish one element from another.

It will be understood that when an element is referred to as being“connected with” another element, the element can be connected with theother element or intervening elements may also be present. In contrast,when an element is referred to as being “directly connected with”another element, there are no intervening elements present.

A singular representation may include a plural representation unless itrepresents a definitely different meaning from the context.

Terms such as “include” or “has” are used herein and should beunderstood that they are intended to indicate an existence of severalcomponents, functions or steps, disclosed in the specification, and itis also understood that greater or fewer components, functions, or stepsmay likewise be utilized.

Such a terminology as a smart controlling device described hereincorresponds to a terminology including all devices configured to performan output operation or a prescribed operation by transmitting data to atleast one or more devices, which are connected or capable of beingconnected based on various inputs including voice, a text, an image, amotion, and the like. The smart controlling device may correspond to allelectronic devices equipped with an artificial intelligence(hereinafter, AI) functionality and/or a voice recognitionfunctionality. For example, an audio device configured to control anoperation of a prescribed device by transmitting a control command tothe prescribed device according to a voice command inputted by a usercan also be included in the smart controlling device. For theunderstanding of the present invention and for clarity, the presentspecification is explained with an example of an audio device (smartspeaker) equipped with AI functionality and the audio device capable ofrecognizing voice, by which the present invention may be non-limited.

FIG. 1 is a schematic diagram for a digital system including a smartcontrolling device according to the present invention.

For clarity, FIG. 1 shows an example of a digital system includingvarious devices including a smart controlling device 110 which isconnected or capable of being connected via a home network. The smartcontrolling device 110 can perform data communication with at least oneor more external devices via short distance or long distancewired/wireless communication to control the at least one or moreexternal devices based on the data communication. For instance, thesmart controlling device 110 can perform a function of a controller inthe digital system shown in FIG. 1. Meanwhile, as shown in FIG. 1, thesmart controlling device 110 can perform data communication with anexternal server 190. In this case, the data communication can beperformed not only with the smart controlling device 110 but also with adifferent device. In this case, the smart controlling device 110 canindirectly perform data communication with the external server 190 viathe different device. The data communication performed with the externalserver or the digital system shown in FIG. 1 can be performed based on acommunication protocol such as WiFi (Wireless-Fidelity), Bluetooth™,Z-wave, Infrared communication, Zigbee, RS, and the like. The presentinvention is not limited to the aforementioned communication protocols.Meanwhile, the data communication can be directly performed betweendevices or can be indirectly performed via such a device as a relay, agap filler, or the like.

For clarity, FIG. 1 shows an example that the smart controlling device110 and other devices 120, 130, 140, 150, 160 included in the digitalsystem belong to the same network. Yet, it is not mandatory that thesmart controlling device 110 and other devices belong to the samenetwork. Meanwhile, referring to FIG. 1, such electronic devices as adigital TV 120, a smartphone 130, an electronic or robot cleaner 140, arefrigerator 150, a washer 160, and the like are illustrated as otherdevices. Yet, the present invention is not restricted by the aboveelectronic devices.

In relation to this, as an example of controlling external devicesaccording to the present invention, the smart controlling device 110 canturn on the power of an air conditioner or control a temperature of theair conditioner. In particular, the smart controlling device 110 canplay a role of a controller configured to control external deviceswithin an IoT (Internet of Things) environment.

FIG. 2 is a block diagram for a smart controlling device and FIG. 3 is adiagram for explaining an exterior of the smart controlling deviceaccording to one embodiment of the present invention.

According to one embodiment of the present invention, the smartcontrolling device includes a signal receiving unit configured toreceive an input signal including a command trigger and a first commanddata and a controller configured to detect and select a display deviceupon the reception of the command trigger. The controller may also beconfigured to control display of a power state of the selected displaydevice to be changed to a first state, as well as to control a responsedata to be outputted on a display of the display device, which isswitched to the first state, in response to the first command data.

The smart controlling device 110 may include a signal receiving unit210, a controller 220, a signal processing unit 230, a memory 240, asignal outputting unit 250, and the like. In this case, the controller220 can include a recognition unit 222, an AI processing unit 224, andthe like. In some embodiments, the recognition unit 222 and the AIprocessing unit 224 can be separately configured apart from thecontroller 220, or they may be implemented in a single configuration.Further, the recognition unit 222 and the AI processing unit 224 may besingularly implemented together in some embodiments and in otherembodiments they may be separately configured, as discussed below anddepicted in FIG. 2 for example.

In an embodiment, the signal receiving unit 210 may receive a voicecommand of a user via a microphone of the terminal (not depicted). Inthis case, for example, the voice command can be divided into such astart signal as “Alice” and an operation signal such as a command, arequest, and the like apart from the start signal. In this discussion,the start signal can be variously referred to as a start data, a voicecommand trigger signal, a command trigger, a voice command trigger data,a trigger signal, a trigger data, or the like. For clarity, the startsignal is commonly referred to as “command trigger.” Unless it isspecifically mentioned, the terms signal and data may be understood torefer to the same information.

Meanwhile, data of an input signal apart from the command trigger isreferred to as a command data. For example, the command data correspondsto an actual command spoken by a speaker (a user). The signal receivingunit 210 is configured to transmit the received voice command to thecontroller 220. In this case, the command trigger can be excluded fromthe transmitted voice command. In some cases, when the command triggeris not inputted by a user, the signal receiving unit 210 may ignore anyreceived signals or may not transmit the signal to the controller 220.Although an embodiment involving voice commands is discussed, thepresent disclosure is not limited to this. In particular, the signalreceiving unit 210 may receive various signals including a text, amotion, a touch, and the like. To this end, the signal receiving unitcan include an interface therefor.

As mentioned in the foregoing description, an embodiment of thecontroller 220 may include the recognizing unit 222, the AI processingunit 224, and the like. The recognizing unit 222 may recognize commandtrigger and command data included in an input signal received from thesignal receiving unit 210 according to a format of the input signal. Forexample, if the input signal received via the signal receiving unit 210is a voice input, the recognizing unit 222 recognizes the input signalusing hardware or software such as a voice recognition function, a tool,a module, an engine, or the like. In this case, the recognizing unit 222converts the voice input signal into a text format based on STT (Soundto Text) (in some cases, TTS (Text to Sound)) and provides the textformat to the AI processing unit 224 to analyze the input signal. The AIprocessing unit 224 analyzes the input signal data provided by therecognizing unit 222. For example, if a command trigger included in thereceived input signal, the recognizing unit 222 or the AI processingunit 224 can detect, select, and control an associated display deviceaccording to the command data input with the command trigger. The AIprocessing unit 224 may analyze the input signal recognized by therecognizing unit 222 and generate response data. In the foregoingdescription, for example, the AI processing unit 224 or the controller220 may be configured to determine whether or not the input signal isvalid, determine an attribute of a trigger or command data, determine atype of the trigger or command data, and the like.

The AI processing unit may include one or more modules configured toperform machine learning, information inference, information perception,natural language processing, and the like.

The AI processing unit 224 may perform machine learning, inference, andprocessing of a vast amount of information (Big data) such asinformation stored in the smart controlling device 110, such as inmemory 240, information on environment surrounding the smart controllingdevice 110, information stored in an external storage capable ofperforming communication, and the like using machine learningtechnology. The AI processing unit 224 may predict executable operationsof at least one or more terminals based on the information learned bymachine learning and may be able to control a terminal to execute anoperation determined to be most appropriate among the predictedoperations.

Machine learning may include collecting and learning a vast amount ofinformation based on at least one or more algorithms and determining andpredicting information based on the learned information. The machinelearning may correspond to operations of identifying a characteristic, arule, a criterion, and the like based on the information, quantifying arelation between information, and predicting new data using a quantifiedpattern.

Algorithms used by the machine learning technology may be based onstatistics. For example, the algorithms may include a decision treeusing a tree structure as a prediction model, a neural network imitatinga neural network structure and function of a creature, geneticprogramming based on an evolution algorithm of a creature, clusteringfor distributing examined examples by such a subset as a cluster, Montecarlo method for calculating a function value with probability via arandomly selected random number, and the like.

As a branch of machine learning technology, a deep learning technologycorresponds to a technology that performs learning, determining, orprocessing information using an artificial neural network algorithm. Theartificial neural network connects layers with each other and may have astructure of forwarding data between layers. The deep learningtechnology can learn a vast amount of information via the artificialneural network using a GPU (graphic processing unit) optimized forparallel arithmetic.

Meanwhile, in order to collect a vast amount of information to which themachine learning technology is applied, the AI processing unit 224 cancollect (sense, monitor, extract, detect, receive) signals, data,information, and the like inputted or outputted to/from configurationelements of the smart controlling device 110. And, the AI processingunit 224 can collect (sense, monitor, extract, detect, receive) data,information, and the like stored in an external storage (e.g., cloudserver) connected with the AI processing unit via communication, orstored in local memory, such as memory 240. More specifically, theinformation collecting may include operations of sensing informationthrough a sensor, extracting information from the memory 240, orreceiving information from the external storage via communication.

In some embodiments, the AI processing unit 224 can sense informationstored in the smart controlling device 110, information on environmentsurrounding the smart controlling device 110, or user informationreceived through a sensor (not depicted). And, the AI processing unit224 can receive a broadcast signal and/or broadcast-related information,a radio signal, a radio data, and the like via a radio communicationunit (not depicted). And, the AI processing unit 224 can receive videoinformation (or signal), audio information (or signal), data, orinformation inputted by a user from an input unit.

The AI processing unit 224 may collect a vast amount of information inthe background in real time, perform learning or training based on thecollected the information, and store the information in the memory 240in a manner of processing the information in an appropriate form (e.g.,knowledge graph, command policy, personalized database, conversationengine, etc.).

If an operation of a terminal is predicted based on the informationlearned using the machine learning technology, the AI processing unit224 may control the configuration elements of the terminal or deliver acontrol command for executing the predicted operation to the controller220 to execute the predicted operation.

Meanwhile, if a specific operation is performed, the AI processing unit224 analyzes history information indicating the performance of thespecific operation via the machine learning technology and may be ableto update previously learned information based on the analyzedinformation. By doing so, the AI processing unit 224 can enhanceaccuracy of information prediction.

In some embodiments, the recognition unit 222 and/or the AI processingunit 224 and the controller 220 can be implemented in the sameconfiguration element. In this case, a function performed by thecontroller 220 described in the present specification can be expressedas being performed by the recognizing unit 222 and/or the AI processingunit 224, and vice versa.

On the contrary, in other embodiments, the recognition unit 222 and/orthe AI processing unit 224 and/or the controller 220 may be implementedas separate configuration elements. In this case, the recognition unit222 and/or the AI processing unit 224 may exchange data with thecontroller 220 to perform various controls in the smart controllingdevice 110. The controller 220 can perform at least one or morefunctions in a terminal or control at least one or more configurationelements of the terminal based on a result determined by the AIprocessing unit 224. Moreover, the AI processing unit 224 may operateunder the control of the controller 220.

The memory 240 may store data necessary for analyzing receivedinformation and processing the information. The memory stores datasupporting various functions of the smart controlling device 110. Thememory 240 can store a plurality of application programs (applications)operating at the smart controlling device 110, data for operations ofthe smart controlling device 110, commands, data (e.g., includingmachine learning data) for operations of the AI processing unit 224, andthe like. The applications may be downloaded from an external server viawired or wireless communication. Meanwhile, an application may be storedin the memory 240 and installed at the smart controlling device 110 toenable the controller 220 perform operations or functions of the smartcontrolling device 110.

The signal processing unit 230 generates a signal according to thecontrol of the controller 220. For example, the generated signal mayinclude the aforementioned control command. Besides, the signalprocessing unit 230 can process various data necessary for the smartcontrolling device 110.

The signal outputting unit 250 transmits a signal processed by thesignal processing unit 230 to an internal or an external deviceaccording to a prescribed output scheme. In this case, the output schemecan be determined or differentiated according to a communicationprotocol, a target device, a type of an output signal, and the like.

Although it is not depicted, if necessary, the smart controlling device110 can further include a communication interface, or the like,necessary for transmitting and receiving a signal/data.

Meanwhile, referring to FIG. 3, the smart controlling device 100 caninclude a user input unit 323, a sound outputting unit 352, and a firstlight outputting unit 354 at an outer side of a body. The user inputunit 323 can be configured to receive a control command from a user andin some embodiments, the user input unit 323 may include a plurality ofuser input units at the smart controlling device. One embodiment mayinclude a first user input unit 323 a, a second user input unit 323 b,and a third user input unit 323 c. Similarly, a plurality of lightoutputting units can be installed in the smart controlling device. Inone embodiment, the plurality of the light outputting units may includea first light outputting unit 354 a and a second light outputting unit354 b. For the purposes of this discussion, the plurality of the userinput units and the plurality of the light outputting units may becollectively referred to as reference characters 323 and 354,respectively.

The body of one embodiment may have a cylindrical shape and have afunction of a speaker. A size of the body can be determined inconsideration of design of the body unit. However, a shape of the bodyunit can be changed in various ways in other embodiments.

The body can include a first region 310 forming a side of a cylinder, asecond region 320 forming a top side of the cylinder, and a third region330 forming a bottom side of the cylinder. The second region 320 and thethird region 330 may or may not have the same area.

The first region 310 can also be referred to as an outer side. Thesecond region 320 and the third region 330 can also be referred to as anouter upper side and an outer bottom side, respectively. Yet, in thefollowing, such terminologies as the first, the second, and third regionare used.

The first region 310 can be equipped with a third user input unit 323 c,a second light outputting unit 354 b, an infrared outputting unit 355,and a sound outputting unit 352. For example, the second lightoutputting unit 354 b and the sound outputting unit 352 can be formed ina manner of being apart from each other. Or, at least a part of thesecond light outputting unit 354 b and the sound outputting unit 352 mayform a layered structure to make the second light outputting unit 354 band the sound outputting unit 352 to be overlapped with each other.

The second light outputting unit 354 b and the sound outputting unit 352can be configured to surround the first region 310 or be positioned atthe first region 310 of the body. Hence, the sound outputting unit 352may be configured to output sound in all directions from the body andthe second light outputting unit 354 b can output light in alldirections from the body.

The third user input unit 323 c can be deployed at the top of the firstregion 310. The third user input unit 323 c can be configured to rotatearound the center of the cylindrical body. In one example, a user mayrotate the third user input unit 323 c to increase or decrease thevolume of the smart controlling device 110.

The infrared outputting unit 355 can be at a location capable ofoutputting an infrared signal in all directions. For example, theinfrared outputting unit may be positioned at the top of the firstregion 310. In another example, the infrared outputting unit may bepositioned at a region configured to rotate at the top of the firstregion 310. Hence, the infrared outputting unit 355 may be able tooutput an infrared signal to an external device located at a randomposition with respect to the body. In other embodiments, various otherlocations of the infrared outputting unit around all positions on thebody are considered.

The second region 320 can be equipped with a display unit 351, first andsecond user input units 323 a/323 b, a first light outputting unit 354a, and a temperature/humidity sensor (not depicted).

The display unit 351 may be positioned at the center of the secondregion 320. The first and the second user input units 323 a/323 b may bepositioned at regions adjacent to the display unit 351 to receive a userinput.

The first and second user input units 323 a/323 b may correspond tophysical or touch sensitive buttons. The first user input unit and thesecond user input unit can be configured to perform different functions.For example, the first user input unit 323 a may correspond to a buttonfor terminating voice recognition and the second user input unit 323 bmay correspond to a button for turning on/off power.

The first light outputting unit 354 a can be configured along an outeredge of the second region 320. For example, if the second region 320corresponds to a circular shape, the first light outputting unit 354 amay be configured as a ring around the edge of the second region.

The light outputting unit 354 includes one or more light sources to emitlight. As an example of the light source, it may use an LED (lightemitted diode). The light source is located at the inner peripheralsurface of the light outputting unit 354 and light emitted from thelight source passes through the light outputting unit 354 and emitslight. The light outputting unit 354 may include a transparent ortranslucent material.

The light outputting unit 354 can output light notification informationrelated to an event at the smart controlling device 110. For example, ifthe smart controlling device 110 performs voice recognition, the lightoutputting unit 354 can output a red light. If the smart controllingdevice 110 is waiting for a modification command, the light outputtingunit 354 can output a yellow light.

The temperature/humidity sensor can be positioned at the second region320 to sense an external temperature and humidity.

Although it is not depicted, the third region 330 can be furtherequipped with a power supply unit for receiving power, an interface fortransceiving data with an external device, such as a network interface,an audio input unit (microphone) for receiving sound, and the like.

In the following, embodiments of a terminal performing datacommunication with the smart controlling device 110 will be discussed.Examples of the terminal include a fixed device such as a networktelevision, a HBBTV (hybrid broadcast broadband TV), a smart TV, an IPTV(internet protocol TV), a PC (personal computer), or the like.Meanwhile, the terminal may also include a display panel only. In otherembodiments, the terminal can be implemented using variousconfigurations and/or combinations (e.g., a display combined with a STB(set-top box)).

The terminal may use a general-purpose OS (operating system), such as“Web OS”, and the like. The terminal can add, delete, modify, and updatevarious services or an application to a general-purpose OS kernel, suchas a Linux kernel, and the like. By doing so, it may be able to providea more user-friendly environment to a user.

The terminal can also be referred to as a display device and thedisclosure considers all embodiments including any terminal deviceequipped with a display. FIG. 4A, FIG. 4B show a digital TV as anexample of the display device and FIG. 5 shows a mobile terminal as adifferent example of the display device. Yet, as mentioned in theforegoing description, the terminal may correspond to any devicesequipped with a display without being restricted to the examples shownin FIG. 4A, FIGS. 4B and 5.

FIG. 4A is a block diagram for a configuration of a digital TV accordingto one embodiment of the present invention.

A digital TV 400 can include a network interface unit 401, a TCP/IPmanager 402, a service delivery manager 403, an SI decoder 404, ademuxer or demultiplexer 405, an audio decoder 406, a video decoder 407,a display unit (display A/V and OSD module) 408, a service controlmanager 409, a service discovery manager 410, an SI & metadata DB 411, ametadata manager 412, a service manager 413, a UI (user interface)manager 414, and the like.

The network interface unit 401 transceives IP packets (Internet Protocolpackets) or IP datagram(s) (hereinafter, “IP packet(s)”) with a server405. For example, the network interface unit 401 can receive a service,an application, contents, etc. from a service provider via a network.

The TCP/IP manager 402 is involved in delivering packets received ortransmitted by the digital TV 400. In particular, the TransmissionControl Protocol/Internet Protocol (TCP/IP) manager 402 is involved indelivering a packet between a source and a destination. The TCP/IPmanager 402 classifies received packet(s) to make the packet(s)correspond to an appropriate protocol and outputs the classifiedpacket(s) to the service delivery manager 403, the service discoverymanager 410, the service control manager 409, the metadata manager 412,and the like.

The service delivery manager 403 is in charge of controlling a receivedservice data. For example, in case of controlling real-time streamingdata, the service delivery manager 403 can use Real-time TransportProtocol/RTP Control Protocol (RTP/RTCP). In case of transmitting thereal-time streaming data using RTP, the service delivery manager 403parses the received data packet according to the RTP and transmits theparsed data packet to the demultiplexer 405 or stores the parsed datapacket in the System Information/Service Information/SignalingInformation (SI) & metadata database (DB) 411 according to the controlof the service manager 413. Subsequently, the service delivery manager403 feeds back the information received from a network to a serviceproviding server using RTCP.

The demultiplexer 405 demultiplexes a received packet into audio, video,SI data, or the like and transmits it to the audio/video decoder 406/407and the SI decoder 404.

The SI decoder 404 decodes demultiplexed SI data such as serviceinformation including PSI (Program Specific Information), PSIP (Programand System Information Protocol), DVB-SI (Digital VideoBroadcasting-Service Information), DTMB/CMMB (Digital TelevisionTerrestrial Multimedia Broadcasting/Coding Mobile MultimediaBroadcasting), and the like. The SI decoder 404 can store decodedservice information in the SI & metadata DB 411. For example, the storedservice information can be used in a manner of being read and outputtedby a corresponding configuration according to a request of a user.

The audio/video decoder 406/407 decodes demultiplexed audio data anddemultiplexed video data, respectively. The decoded audio data and thedecoded video data are provided to a user via the display unit 408.

An application manager, for example, includes a UI manager 414 and aservice manager 413 and can perform a function of a control unit of thedigital TV 400. The application manager manages the overall state of thedigital TV 400, provides a user interface (UI), and manages a differentmanager.

The UI manager 414 provides GUI (Graphic User Interface)/UI to a userusing OSD (On Screen Display), receives a key input from the user, andperforms a device operation according to the input. For example, if theUI manager 414 receives a key input for selecting a channel from a user,the UI manager transmits the key input signal to the service manager413.

The service manager 413 controls managers related to a service such asthe service delivery manager 403, the service discovery manager 410, theservice control manager 409, the metadata manager 412, and the like. Theservice manager 413 generates a channel map and controls channelselection and the like using the generated channel map according to thekey input received from the UI manager 414. The service manager 413 setsan audio/video PID (Packet identifier) of a channel, which is selectedby receiving service information from the SI decoder 404, to thedemultiplexer 405. The PID can be used for the aforementionedde-multiplexing procedure. The demultiplexer 405 filters audio data,video data, and SI data using the PID.

The service discovery manager 410 provides a user with informationnecessary for selecting a service provider providing a service. If asignal for selecting a channel is received from the service manager 413,the service discovery manager 410 discovers a service using theinformation.

The service control manager is in charge of selecting and controlling aservice. For example, if a user selects a live broadcasting servicesimilar to a legacy broadcasting scheme, the service control manager 409may use IGMP (Internet Group Management Protocol) or RTSP. If the userselects such a service as VOD (Video on Demand), the service controlmanager selects and controls the service using RTSP. The RTSP protocolcan provide a trick mode to real-time streaming. The service controlmanager 409 can initiate and manage a session via an IMS gateway 430using IMS (IP Multimedia Subsystem), SIP (Session Initiation Protocol).The aforementioned protocols are just an embodiment. Depending on animplementation example, it may be able to use a different protocol.

The metadata manager 412 manages a metadata associated with a serviceand stores the metadata in the SI & metadata DB 411.

The SI & metadata DB 411 stores information necessary for selectingservice information decoded by the SI decoder 404, metadata managed bythe metadata manager 412, and a service provider provided by the servicediscovery manager 410. The SI & metadata DB 411 can store set-up datafor a system, and the like. The SI & metadata DB 411 can be implementedusing a non-volatile RAM (NVRAM), a flash memory, and the like.

Meanwhile, the IMS gateway 430 corresponds to a gateway includingfunctions necessary for accessing an IMS-based IPTV service.

FIG. 4B is a block diagram for a configuration of a digital TV accordingto a different embodiment of the present invention.

Referring to FIG. 4B, a digital TV 400 can include a broadcast receivingunit 435, an external device interface unit 448, a storing unit 455, auser interface unit 480, a control unit 450, a display unit 460, anaudio output unit 465, a power supply unit 470, and a capturing unit(not depicted). In this case, the broadcast receiving unit 435 caninclude at least one tuner 442, a demodulating unit 444, and a networkinterface unit 446. In some cases, the broadcast receiving unit 435 mayinclude the tuner 442 and the demodulating unit 444 only except thenetwork interface unit 446, or vice versa. Although it is not depicted,the broadcast receiving unit 435 may include a multiplexer to multiplexa signal demodulated by the demodulating unit 444 via the tuner 442 witha signal received through the network interface unit 446. Besides,although it is not depicted as well, the broadcast receiving unit 435may include a demultiplexer to de-multiplex the multiplexed signal, thedemodulated signal, or the signal went through the network interfaceunit 446.

The tuner 442 receives an RF (Radio Frequency) broadcasting signal bytuning a channel selected by a user or all channels stored in advanceamong RF broadcasting signals received via an antenna. The tuner 442converts the received RF broadcasting signal into an IF (IntermediateFrequency) signal or a baseband signal. For example, if the received RFbroadcasting signal corresponds to a digital broadcasting signal, the RFbroadcasting signal is converted into a digital IF signal (DIF). If thereceived RF broadcasting signal corresponds to an analog broadcastingsignal, the RF broadcasting signal is converted into an analog basebandimage or an audio signal (CVBS/SIF). In particular, the tuner 442 canprocess both a digital broadcasting signal and an analog broadcastingsignal. The analog baseband image or the audio signal (CVBS/SIF)outputted from the tuner 442 can be directly inputted to the controlunit 450. The tuner 442 can receive an RF broadcasting signal of asingle carrier or multiple carriers. Meanwhile, the tuner 442sequentially tunes and receives RF broadcasting signals of allbroadcasting channels, which are stored via a channel remember function,among RF broadcasting signals received via an antenna and may be able toconverts the received RF broadcasting signals into a DIF (DigitalIntermediate Frequency or baseband signal).

The demodulating unit 444 receives the digital IF signal (DIF) convertedby the tuner 442, demodulates the signal, and performs channel decoding.To this end, the demodulating unit 444 can be equipped with a Trellisdecoder, a de-interleaver, a Reed-Solomon decoder, a convolutiondecoder, a de-interleaver, a Reed-Solomon decoder, or the like. Thedemodulating unit 444 performs demodulation and channel decoding and maybe then able to output a stream signal (TS (transport stream)). In thiscase, the stream signal may correspond to a signal that a video signal,an audio signal, and a data signal are multiplexed. For example, thestream signal may correspond to MPEG-2 TS that a video signal of MPEG-2standard and an audio signal of Dolby AC-3 standard are multiplexed. Thestream signal outputted from the demodulating unit 444 can be inputtedto the control unit 450. The control unit 450 controls de-multiplexing,video/audio signal processing, and the like, controls an image to beoutputted via the display unit 460, and controls audio to be outputtedvia the audio output unit 465.

The external device interface unit 448 provides interfacing environmentbetween the digital TV 400 and various external devices. To this end,the external device interface unit 448 can include an A/V input/outputunit (not depicted) or a wireless communication unit (not depicted). Theexternal device interface unit 448 can be connected with an externaldevice such as a DVD (Digital Versatile Disk), a Blu-ray, a gamingdevice, a camera, a camcorder, a computer (notebook), a tablet PC, asmartphone, a Bluetooth device, a cloud, and the like in wired/wireless.The external device interface unit 448 delivers a signal including datasuch as an image, a video, audio, and the like inputted via an externaldevice connected with the external device interface unit to the controlunit 450 of the digital TV 400. The control unit 450 can control a datasignal including a processed image, a video, audio, and the like to beoutputted to the connected external device. To this end, the externaldevice interface unit 448 can further include an A/V input/output unit(not depicted) or a wireless communication unit (not depicted).

The A/V input/output unit can include a USB terminal, a CVBS terminal, acomponent terminal, an S-video terminal (analog), a DVI terminal, a HDMIterminal, an RGB terminal, a D-SUB terminal, and the like to input animage signal and an audio signal of an external device to the digital TV400.

The wireless communication unit can perform short range wirelesscommunication with a different digital device. For example, the digitalTV 400 can be connected with a different digital device according to acommunication protocol such as Bluetooth, RFID, infrared communication(IrDA), UWB, Zigbee, DLNA, or the like.

The external device interface unit 448 is connected with a set-top box(STB) via at least one of the aforementioned terminals and performs aninput/output operation with the set-top box (STB). Meanwhile, theexternal device interface unit 448 receives an application or a list ofapplications of an external device adjacent to the external deviceinterface unit and delivers the application or the list of applicationsto the control unit 450 or the storing unit 455.

The network interface unit 446 provides an interface to connect thedigital TV 400 with a wired/wireless network including the Internet. Inorder to connect the digital TV with a wired network, for example, thenetwork interface unit 446 can include an Ethernet terminal. In order toconnect the digital TV with a wireless network, for example, the networkinterface unit 446 may use WLAN, Wibro, Wimax, HSDPA communicationstandard, or the like. The network interface unit 446 can transceive adata with a different user or a different digital device via a connectednetwork or a different network linked with the connected network. Inparticular, partial content data stored in the digital TV can betransmitted to a user or a digital device selected from users or digitaldevices registered at the digital device in advance. Meanwhile, thenetwork interface unit 446 can access a prescribed web page via aconnected network or a different network linked with the connectednetwork. In particular, the network interface unit accesses a prescribedweb page via a network and may be able to transmit or receive a datawith a corresponding server. Besides, the network interface unit canreceive contents or data provided by a content provider or a networkprovider. In particular, the network interface unit can receive contentsprovided by the content provider or the network provider such as amovie, advertisement, game, VOD, a broadcasting signal, and the like andinformation related to the contents via a network. And, the networkinterface unit can receive update information and an update file offirmware provided by a network manager. And, the network interface unitcan transmit data to internet or content provider or a network provider.And, the network interface unit 446 can select an application preferredby the network interface unit from among disclosed applications via anetwork and receive the application.

The storing unit 455 can store a program for processing and controllinga signal of the control unit 450 and a signal-processed video, audio, ora data signal. And, the storing unit 455 may perform a function oftemporarily storing video, audio, or a data signal inputted from theexternal device interface unit 448 or the network interface unit 446.The storing unit can store information on a prescribed broadcastingchannel through a channel remember function. The storing unit 455 canstore an application or a list of applications inputted from theexternal device interface unit 448 or the network interface unit 446.And, the storing unit 455 can store various platforms described in thefollowing. For example, the storing unit 455 can include at least onestoring medium selected from the group consisting of a flash memory typememory, a hard disk type memory, a multimedia card micro type memory, acard type memory (e.g., SD or XD memory), RAM, and ROM (EEPROM, etc.).The digital TV 400 plays content files (a video file, a still imagefile, a music file, a document file, an application file, etc.) storedin the storing unit 455 and provides the files to a user. Although FIG.4b shows an embodiment that the storing unit 455 and the control unit450 are separately installed, by which the present invention may benon-limited. In particular, the storing unit 455 can be included in thecontrol unit 450.

The user input interface unit 480 delivers a signal inputted by a userto the control unit 450 or delivers a signal of the control unit 450 toa user. For example, the user input interface unit 480 receives acontrol signal such as power on/off, channel selection, screenconfiguration, etc. from a remote control device 490 and processes thecontrol signal according to various communication schemes including anRF communication scheme, an infrared communication scheme, and the like.Or, the user input interface unit 480 can transmit a control signalreceived from the control unit 450 to the remote control device 490. Theuser input interface unit 480 can deliver a control signal inputted by alocal key (not depicted) such as a power key, a channel key, a volumekey, a setting key, and the like to the control unit 450. The user inputinterface unit 480 can deliver a control signal inputted from a sensingunit (not depicted) configured to sense a gesture of a user to thecontrol unit 450 or transmit a signal of the control unit 450 to thesensing unit (not depicted). In this case, the sensing unit (notdepicted) can include a touch sensor, a voice sensor, a location sensor,an operation sensor, and the like.

The control unit 450 can generate and output a signal for outputtingvideo or audio by de-multiplexing a stream inputted via the tuner 442,the demodulating unit 444, or the external device interface unit 448 orprocessing demultiplexed signals. A video signal processed by thecontrol unit 450 is inputted into the display unit 460 and can bedisplayed as a video corresponding to the video signal. And, the videosignal processed by the control unit 450 can be inputted into anexternal output device via the external device interface unit 448. Anaudio signal processed by the control unit 450 can be inputted into anexternal output device via the external device interface unit 448.Although it is not depicted in FIG. 4B, the control unit 450 can includea de-multiplexing unit, an image processing unit, and the like.

The control unit 450 can control overall operation of a digital TV. Forexample, the control unit 450 controls the tuner 442 to tune RFbroadcasting corresponding to a channel selected by a user or a channelstored in advance. The control unit 450 can control the digital TVaccording to a user command inputted via the user input interface unit480 or an internal program. In particular, the control unit accesses anetwork and may be able to enable a user to download an application or alist of application preferred by the user to the digital TV. Forexample, the control unit 450 controls the tuner 442 to input a signalof a channel which is selected according to a channel selection commandreceived via the user input interface unit 480. The control unitprocesses video, audio, or a data signal of the selected channel. Thecontrol unit 450 controls the display unit 460 or the audio output unit465 to output information on the channel selected by a user togetherwith a processed video signal or an audio signal.

As a different example, the control unit 450 controls a video signal oran audio signal of an external device (e.g., a camera or a camcorder)inputted via the external device interface unit 448 to be outputtedthrough the display unit 460 or the audio output unit 465 according toan external device image playback command received via the user inputinterface unit 480. Meanwhile, the control unit 450 can control thedisplay unit 460 to display an image. For example, the control unit cancontrol a broadcasting image inputted via the tuner 442, an externalinput image inputted via the external device interface unit 448, animage inputted via the network interface unit, or an image stored in thestoring unit 455 to be displayed on the display unit 460. In this case,the image displayed on the display unit 460 may correspond to a stillimage or a video. The image may correspond to a 2D image or a 3D image.And, the control unit 450 can control content to be played. In thiscase, the content may correspond to content stored in a digital TV,received broadcasting content, or external input content inputted fromthe external. The content may correspond to a broadcasting image, anexternal input image, an audio file, a still image, an accessed webscreen, or a document file. Meanwhile, in case of entering anapplication watch item, the control unit 450 can control an applicationor a list of applications capable of being downloaded from a digital TVor an external network to be displayed. The control unit 450 can controlan application downloaded from an external network together with varioususer interfaces to be installed and executed. And, the control unit 450can control an image related to an executed application to be displayedon the display unit 460 according to the selection of a user.

Meanwhile, although it is not depicted in the drawing, the digital TVcan further include a channel browsing processing unit configured togenerate a thumbnail image in response to a channel signal or anexternal input signal. The channel browsing processing unit receives aninput of a stream signal (TS) outputted by the demodulating unit 444 oran input of a stream signal outputted by the external device interfaceunit 448, extracts an image from the inputted stream signal, andgenerates a thumbnail image. The generated thumbnail image can beinputted to the control unit 450 as it is or can be inputted to thecontrol unit in a manner of being encoded. The control unit 450 candisplay a thumbnail list including a plurality of thumbnail images onthe display unit 460 using the inputted thumbnail image. Meanwhile, thethumbnail images included in the thumbnail list can be sequentially orsimultaneously updated. Hence, a user can conveniently identify contentsof a plurality of broadcasting channels.

The display unit 460 converts an image signal, a data signal, and an OSDsignal processed by the control unit 450, or an image signal, a datasignal, and the like received from the external device interface unit448 into R, G, and B signal to generate a driving signal. The displayunit 460 may correspond to a PDP (Plasma Display Panel), an LCD (LiquidCrystal Display), an OLED (Organic Light Emitting Diode), a flexibledisplay, a 3D display, or the like. Meanwhile, the display unit 460 canbe used as an input device as well as an output device in a manner ofbeing configured by a touch screen. The audio output unit 465 receives asignal (e.g., a stereo signal, 3.1 channel signal, or 5.1 channelsignal) from the control unit 450 and outputs the signal as audio. Theaudio output unit 465 can be implemented by a speaker of various types.

Meanwhile, in order to sense a gesture of a user, as mentioned in theforegoing description, a sensing unit (not depicted) including at leastone selected from the group consisting of a touch sensor, an audiosensor, a location sensor, and an operation sensor can be furtherinstalled in the digital TV. A signal sensed by the sensing unit (notdepicted) can be delivered to the control unit 450 via the user inputinterface unit 480. Meanwhile, a capturing unit (not depicted)configured to capture a user can be further installed in the digital TV.Information on an image captured by the capturing unit (not depicted)can be inputted into the control unit 450. The control unit 450 cansense a gesture of a user by combining an image captured by thecapturing unit (not depicted) and a signal sensed by the sensing unit(not depicted).

The power supply unit 470 supplies power to the digital TV. Inparticular, the power supply unit can supply power to the control unit450 capable of being implemented in a form of SoC (System on Chip), thedisplay unit 460 for displaying an image, and the audio output unit 465for outputting audio. To this end, the power supply unit 490 can includea converter (not depicted) configured to convert AC power into DC power.Meanwhile, for example, if the display unit 460 is implemented by aliquid crystal panel including a plurality of backlight lamps, thedisplay unit can further include an inverter (not depicted) capable ofperforming a PWM (Pulse Width Modulation) operation for brightnessvariableness or a dimming operation.

The remote control device 490 transmits a user input to the user inputinterface unit 480. To this end, the remote control device 490 may useBluetooth, RF communication, Infrared (IR) communication, UWB, a Zigbeescheme, or the like. The remote control device 490 receives video,audio, or a data signal outputted by the user input interface unit 480and displays the video, the audio, or the data signal on the remotecontrol device 490. Or, the remote control device may output audio orvibration.

Besides, the digital TV according to the present invention may omit apartial component from among the displayed components or can furtherinclude a component not shown in the drawing if necessary. Meanwhile,unlike what is mentioned earlier, the digital TV may receive contentsvia the network interface unit or the external device interface unit andplay the contents while not including the tuner and the demodulatingunit.

Although it is not depicted, the aforementioned control unit can includea de-multiplexing unit, an image processing unit, an OSD generatingunit, a mixer, a frame rate converter (FRC), and a formatter. And, thecontrol unit can further include an audio processing unit and a dataprocessing unit.

The de-multiplexing unit demultiplexes an inputted stream. For example,the de-multiplexing unit can de-multiplex an inputted MPEG-2 TS image,audio and a data signal. In this case, the stream signal inputted to thede-multiplexing unit may correspond to a stream signal outputted fromthe tuner, the demodulating unit, or the external device interface unit.

The image processing unit performs image processing on a demultiplexedimage signal. To this end, the image processing unit can include animage decoder and a scaler. The image decoder decodes a demultiplexedimage signal and the scaler scales resolution of the decoded imagesignal to display the image signal on the display unit. The imagedecoder can support various specifications. For example, if an imagesignal is encoded by MPEG-2 specification, the image decoder performs afunction of an MPEG-2 decoder. If an image signal is encoded by a DMB(Digital Multimedia Broadcasting) scheme or H.264/H.265 specification,the image decoder can perform a function of H.264/H.265 decoder.Meanwhile, an image signal decoded by the image processing unit isinputted into the mixer.

The OSD generating unit generates OSD data according to a user input orautonomously. For example, the OSD generating unit generates data todisplay various data on a screen of the display unit in a graphic/textform/format based on a control signal of the user input interface unit.The generated OSD data includes various data including a user interfacescreen of the digital TV, various menu screens, a widget, an icon,viewing rate information, and the like. The OSD generating unit maygenerate data to display subtitles of a broadcasting image orbroadcasting information based on an EPG (Electronic Program Guide).

The mixer mixes the OSD data generated by the OSD generating unit andthe image signal processed by the image processing unit and provides itto the formatter. If the decoded image signal and the OSD data aremixed, OSD is displayed on a broadcasting image or an external inputimage in a manner of being overlaid.

The frame rate converter (FRC) converts a frame rate of an inputtedimage. For example, the frame rate converter can convert an inputtedframe rate of 60 Hz into a frame rate of 120 Hz or 240 Hz according toan output frequency of the display unit. As mentioned in the foregoingdescription, a frame rate can be converted in various methods. Forexample, when the frame rate converter converts a frame rate of 60 Hzinto a frame rate of 120 Hz, it may insert the same first frame betweenthe first frame and a second frame or it may insert a third frame, whichis anticipated from the first frame and the second frame, between thefirst frame and the second frame. As a different example, when the framerate converter converts a frame rate of 60 Hz into a frame rate of 240Hz, it may be able to insert the same frame or 3 anticipated framesbetween the legacy frames. Meanwhile, in case of not performing separateframe conversion, it may bypass the frame rate converter.

The formatter changes an output of the frame rate converter, which isinputted to the formatter, in accordance with an output format of thedisplay unit. For example, the formatter can output R, G, and B datasignal. The R, G, and B data signal can be outputted by a LVDS (LowVoltage Differential Signal) or mini-LVDS. If the output of the framerate converter corresponds to a 3D image signal, the formatter outputs a3D format output in accordance with the output format of the displayunit. By doing so, it may be able to support a 3D service via thedisplay unit.

Meanwhile, the audio processing unit included in the control unit canperform audio processing on a demultiplexed audio signal. The audioprocessing unit can be configured to process various audio formats. Forexample, if an audio signal is encoded by a format such as MPEG-2,MPEG-4, AAC, HE-AAC, AC-3, BSAC, or the like, the audio signal can beprocessed by including a decoder corresponding to the format. The audioprocessing unit included in the control unit can process base, treble,volume control, and the like. The data processing unit included in thecontrol unit can perform data processing on a demultiplexed data signal.For example, although a demultiplexed data signal is encoded, the dataprocessing unit can decode the signal. In this case, the encoded datasignal may correspond to EPG information including start time of abroadcasting program broadcasted on each channel, end time, and thelike.

Meanwhile, the aforementioned digital TV corresponds to an exampleaccording to the present invention. Each of the components can beintegrated, added, or omitted according to the specification of thepractically implemented digital TV. In particular, if necessary, two ormore components can be integrated into one component or one componentcan be divided into two or more components. A function performed by eachblock explains an embodiment of the present invention. The scope ofright of the present invention is not restricted by a specific operationor a device. Meanwhile, the digital device may correspond to an imagesignal processing device that performs signal processing on an imagestored in the device or an inputted image. As a different example of theimage signal processing device, a set-top box from which the displayunit 460 and the audio output unit 465 shown in FIG. 4A are excluded,the aforementioned DVD player, a Blue-ray player, a gaming device, acomputer, and the like may exist.

It will be apparent to those skilled in the art that the configurationdiscussed and shown in FIGS. 4A to 4B may be applicable to a mobileterminal as described in the following.

For example, a mobile terminal according to the present specificationcan include a voice recognition speaker, a mobile phone, a smartphone, alaptop computer, a digital broadcasting terminal, a PDA (PersonalDigital Assistant), a PMP (Portable Multimedia Player), a navigation, aslate PC, a tablet PC, ultra book, a wearable device (e.g., a smartwatch, a smart glass, a HMD (head mounted display)), and the like.

In the present specification, as an example of a terminal performingdata communication with a smart controlling device 110 according to thepresent invention, a mobile terminal is explained in the following.

FIG. 5 is a block diagram to describe a mobile terminal as one exampleof a source device according to one embodiment of the present invention.

The mobile terminal 500 includes components such as a wirelesscommunication unit 510, an input unit 520, a sensing unit 540, an outputunit 550, an interface unit 560, a memory 570, a controller 580, and apower supply unit 590. It is understood that implementing all of theillustrated components shown in FIG. 5 is not a requirement, and thatgreater or fewer components may alternatively be implemented.

In particular, the wireless communication unit 510 among the componentsmay typically include at least one or more modules which permitcommunications such as wireless communications between the mobileterminal 500 and a wireless communication system, communications betweenthe mobile terminal 500 and another mobile terminal, communicationsbetween the mobile terminal 500 and an external server. Further, thewireless communication unit 510 typically includes at least one or moremodules which connect the mobile terminal 500 to at least one or morenetworks.

The wireless communication unit 510 may include at least one of abroadcast receiving module 511, a mobile communication module 512, awireless Internet module 513, a short-range communication module 514,and a location information module 515.

The input unit 520 may include a camera (or a video or image input unit)521 for a video or image signal input, a microphone 522, which is onetype of audio input device for inputting an audio signal, and a userinput unit 523 (for example, a touch key, a push key (or, a mechanicalkey), etc.) for allowing a user to input information. Data (for example,audio, video, image, and the like) is obtained by the input unit 520 andmay be analyzed and processed into a user's control command bycontroller 580.

The sensing unit 540 may be typically implemented using at least one ormore sensors configured to sense internal information of the mobileterminal, the surrounding environment of the mobile terminal, userinformation, and the like. For example, the sensing unit 540 includes aproximity sensor 541 and an illumination sensor 542. If desired, thesensing unit 540 may alternatively or additionally include other typesof sensors or devices, such as a touch sensor, an acceleration sensor, amagnetic sensor, a gravity sensor (G-sensor), a gyroscope sensor, amotion sensor, an RGB sensor, an infrared (IR) sensor, a finger scansensor, a ultrasonic sensor, an optical sensor (for example, camera521), a microphone 522, a battery gauge, an environment sensor (forexample, a barometer, a hygrometer, a thermometer, a radiation detectionsensor, a thermal sensor, and a gas sensor, among others), and achemical sensor (for example, an electronic nose, a health care sensor,a biometric sensor, and the like), to name a few. The mobile terminal500 may be configured to utilize information obtained from the sensingunit 540, and in particular, information obtained from at least one ormore sensors of the sensing unit 540, and combinations thereof.

The output unit 550 may be typically configured to output various typesof information, such as audio, video, tactile output, and the like. Theoutput unit 550 includes a display unit 551, an audio output unit (ormodule) 552, a haptic module 553, and an optical output unit (or module)554. The display unit 551 may have an inter-layered structure or anintegrated structure with a touch sensor in order to embody atouchscreen. The touchscreen may provide an output interface between themobile terminal 500 and a user, as well as function as the user inputunit 523 which provides an input interface between the mobile terminal500 and the user.

The interface unit 560 serves as an interface with various types ofexternal devices that can be coupled to the mobile terminal 500. Theinterface unit 560, for example, may include any of wired or wirelessheadset ports, external power supply ports, wired or wireless dataports, memory card ports, ports for connecting a device having anidentification module, audio input/output (I/O) ports, video I/O ports,earphone ports, and the like. In some cases, the mobile terminal 500 mayperform assorted control functions associated with a connected externaldevice, in response to the external device being connected to theinterface unit 560.

The memory 570 is typically implemented to store data to support variousfunctions or features of the mobile terminal 500. For instance, thememory 570 may be configured to store application programs (or programs)executed or run in the mobile terminal 500, data or instructions (orcommands) for operations of the mobile terminal 500, and the like. Someof these application programs may be downloaded from an external servervia wireless communication. Other application programs may be installedwithin the mobile terminal 500 at the time of manufacturing or shipping,which is typically the case for basic functions of the mobile terminal500 (for example, receiving a call, placing a call, receiving a message,sending a message, and the like). It is common for application programsto be stored or saved in the memory 570, installed on the mobileterminal 500, and executed by the controller 580 to perform an operation(or function) of the mobile terminal 500.

The controller 580 may typically functions to control overall operationsof the mobile terminal 500, in addition to the operations associatedwith the application programs. The controller 580 may provide or processinformation or functions appropriate for a user by processing signals,data, information and the like, which are input or output by the variouscomponents mentioned in the foregoing description, or activatingapplication programs stored in the memory 570.

Moreover, in order to execute or run the application program saved inthe memory 570, the controller 580 can control some or all of thecomponents illustrated in FIG. 5. Furthermore, in order to execute orrun the application program, the controller 580 may activate at leasttwo of the components included in the mobile terminal 500 in a mannerthat the at least two components are combined together.

The power supply unit 590 can be configured to receive external power orprovide internal power in order to supply appropriate power required foroperating elements and components included in the mobile terminal 500.The power supply unit 590 may include a battery, and the battery may beconfigured to be embedded in the terminal body, or configured to bedetachable from the terminal body.

At least some of the components mentioned in the foregoing descriptioncan operate cooperatively to implement operations, controls orcontrolling methods of the mobile terminal according to variousembodiments mentioned in the following description. Moreover, theoperations, controls and controlling methods of the mobile terminal maybe implemented on the mobile terminal by running or executing at leastone application program saved in the memory 560.

The broadcast reception module 511 of the radio communication unit 510is typically configured to receive a broadcast signal and/or broadcastassociated information from an external broadcast managing server via abroadcast channel. The broadcast channel may include a satellitechannel, a terrestrial channel, or both. In some embodiments, two ormore broadcast reception modules 511 may be utilized to facilitatesimultaneously receiving of two or more broadcast channels, or tosupport switching among broadcast channels.

The broadcast managing server may mean a server which generates andtransmits a broadcast signal and/or broadcast associated information, ora server which receives a pre-generated broadcast signal and/orbroadcast associated information, and sends such items to the terminal.The broadcast signal may be implemented using any of a TV broadcastsignal, a radio broadcast signal, a data broadcast signal, andcombinations thereof, among others. The broadcast signal in some casesmay further include a data broadcast signal combined with a TV or radiobroadcast signal.

The broadcast signal may be encoded according to any of a variety oftechnical standards or broadcasting methods (for example, InternationalOrganization for Standardization (ISO), International ElectrotechnicalCommission (IEC), Digital Video Broadcast (DVB), Advanced TelevisionSystems Committee (ATSC), and the like) for transmission and receptionof digital broadcast signals. The broadcast receiving module 111 canreceive the digital broadcast signals using a method appropriate for thetransmission method utilized.

Examples of the broadcast associated information may include informationassociated with a broadcast channel, a broadcast program, a broadcastevent, a broadcast service provider, or the like. The broadcastassociated information may also be provided via a mobile communicationnetwork, and in this case, received by the mobile communication module512.

The broadcast associated information may be implemented in variousformats. For instance, the broadcast associated information may includean Electronic Program Guide (EPG) of Digital Multimedia Broadcasting(DMB), an Electronic Service Guide (ESG) of Digital VideoBroadcast-Handheld (DVB-H), and the like. Broadcast signals and/orbroadcast associated information received via the broadcast receivingmodule 111 may be stored in a suitable device, such as a memory 570.

The mobile communication module 512 can transmit and/or receive wirelesssignals to and from at least one or more network entities. Typicalexamples of a network entity include a base station, an externalterminal, a server, and the like. Such network entities form part of amobile communication network, which is constructed according totechnical standards or communication methods for mobile communications(for example, Global System for Mobile Communication (GSM), CodeDivision Multi Access (CDMA), CDMA2000 (Code Division Multi Access2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-DataOnly), Wideband CDMA (WCDMA), High Speed Downlink Packet access (HSDPA),HSUPA (High Speed Uplink Packet Access), Long Term Evolution (LTE),LTE-A (Long Term Evolution-Advanced), and the like).

The wireless signals transmitted and/or received via the mobilecommunication module 512 may include audio call signals, video(telephony) call signals, or various formats of data to supportcommunication of text and multimedia messages.

The wireless Internet module 513 is configured to facilitate wirelessInternet access. This module may be internally or externally coupled tothe mobile terminal 500. The wireless Internet module 513 may beconfigured to transmit and/or receive wireless signals via communicationnetworks according to wireless Internet technologies.

Examples of such wireless Internet technologies include Wireless LAN(WLAN), Wireless Fidelity (Wi-Fi), Wi-Fi Direct, Digital Living NetworkAlliance (DLNA), Wireless Broadband (WiBro), Worldwide Interoperabilityfor Microwave Access (WiMAX), High Speed Downlink Packet Access (HSDPA),HSUPA (High Speed Uplink Packet Access), Long Term Evolution (LTE),LTE-A (Long Term Evolution-Advanced), and the like. The wirelessInternet module 513 may transmit/receive data according to at least oneof such wireless Internet technologies, and other Internet technologiesas well.

In some embodiments, when the wireless Internet access is implementedaccording to, for example, WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE,LTE-A and the like, as part of a mobile communication network, thewireless Internet module 513 performs such wireless Internet access. Assuch, the Internet module 513 may cooperate with, or function as, themobile communication module 512.

The short-range communication module 514 is configured to facilitateshort-range communications. Suitable technologies for implementing suchshort-range communications include Bluetooth, Radio FrequencyIDentification (RFID), Infrared Data Association (IrDA), Ultra-WideBand(UWB), ZigBee, Near Field Communication (NFC), Wireless-Fidelity(Wi-Fi), Wi-Fi Direct, Wireless USB (Wireless Universal Serial Bus), andthe like. The short-range communication module 514 in general supportswireless communications between the mobile terminal 500 and a wirelesscommunication system, communications between the mobile terminal 500 andanother mobile terminal 500, or communications between the mobileterminal 500 and a network where another mobile terminal 500 (or anexternal server) is located, via wireless area networks. One example ofthe wireless area networks is a wireless personal area networks.

In some embodiments, another mobile terminal (which may be configuredsimilarly to mobile terminal 500) may be a wearable device, for example,a smart watch, a smart glass or a head mounted display (HIVID), which isable to exchange data with the mobile terminal 500 (or otherwiseinterwork with the mobile terminal 500). The short-range communicationmodule 514 may sense or recognize the wearable device, and permitcommunication between the wearable device and the mobile terminal 500.In addition, when the sensed wearable device is a device which isauthenticated to communicate with the mobile terminal 500, thecontroller 580, for example, may cause transmission of data processed inthe mobile terminal 500 to the wearable device via the short-rangecommunication module 514. Hence, a user of the wearable device may usethe data processed in the mobile terminal 500 on the wearable device.For example, when a call is received in the mobile terminal 500, theuser may answer the call using the wearable device. Also, when a messageis received in the mobile terminal 500, the user can check the receivedmessage using the wearable device.

The position information module 515 is generally configured to detect,calculate, derive or otherwise identify a location of the mobileterminal. As a representative example, the position information module515 includes a Global Position System (GPS) module, a Wi-Fi module, orboth. For instance, if utilizing a GPS (global positioning system)module, the mobile terminal can obtain a location of the mobile terminalusing a signal sent by a GPS satellite. For another instance, ifutilizing the Wi-Fi module, the mobile terminal can obtain its locationbased on information of a wireless AP (access point) configured totransmit or receive wireless signals to or from the Wi-Fi module. Ifdesired, the position information module 515 may alternatively oradditionally function with any of the other modules of the radiocommunication unit 510 to obtain data related to the position of themobile terminal. The position information module 515 may include amodule to obtain a location (or a current location) of the mobileterminal and is non-limited to a module for directly calculating orobtaining a location of the mobile terminal.

The input unit 520 may be configured to permit various types of inputsto the mobile terminal 500. Examples of such inputs include image (orvideo) information (or signal), audio information (or signal), data, anduser input. For the input of the image or video information, the mobileterminal 500 may include one or a plurality of cameras 521. Such cameras521 may process image frames of still pictures or video obtained byimage sensors in a video call mode or an image capture mode. Theprocessed image frames can be displayed on the display unit 551 orstored in memory 560. In some cases, the cameras 521 provided to themobile terminal 500 may be arranged in a matrix configuration to permita plurality of images having various angles or focal points to be inputto the mobile terminal 100. As another example, the cameras 521 may bedisposed in a stereoscopic arrangement to acquire left and right imagesfor implementing a stereoscopic image.

The microphone 522 processes an external sound signal into audio data.Such audio data can be variously utilized in accordance with a function(or application program) currently run in the mobile terminal 500. Ifdesired, the microphone 522 may include assorted noise removingalgorithms to remove unwanted noise generated in the course of receivingthe external sound data.

The input unit 520 is a component that permits input by a user. Ifinformation is inputted through the input unit 520, the controller 580can control an operation of the mobile terminal 500 to correspond to theinputted information. The input unit 520 may include mechanical inputmeans (for example, a mechanical key, a button disposed on a frontand/or rear surface or a side surface of the mobile terminal 500, a domeswitch, a jog wheel, a jog switch, and the like), or a touch-sensitiveinput means, among others. As one example, touch-sensitive input meansmay be a virtual key, a soft key, or a visual key, which is displayed ona touchscreen through software processing, or a touch key which isdisposed on the mobile terminal 500 at a location that is other than thetouchscreen. On the other hand, the virtual key or the visual key may bedisplayed on the touchscreen in various shapes, for example, graphic,text, icon, video, or a combination thereof.

Meanwhile, the sensing unit 540 is generally configured to sense atleast one of internal information of the mobile terminal, surroundingenvironment information of the mobile terminal, user information, andthe like and then generates a sensing corresponding to the sensedinformation. Based on the sensing signal, the controller 580 generallycooperates with the sensing unit 540 to control operation of the mobileterminal 500 or execute data processing, a function or an operationassociated with an application program installed on the mobile terminal.The sensing unit 540 may be implemented using any of a variety ofrepresentative sensors, some of which will now be described in moredetail as follows.

First of all, the proximity sensor 541 may include a sensor to sense apresence or absence of an object approaching a surface, or an objectdisposed near a surface, by using an electromagnetic field, infraredrays, or the like without a mechanical contact. The proximity sensor 541may be arranged at an inner region of the mobile terminal covered by thetouchscreen, or near the touchscreen.

The proximity sensor 541, for example, may include any of a transmissivetype photoelectric sensor, a direct reflective type photoelectricsensor, a mirror reflective type photoelectric sensor, a high-frequencyoscillation proximity sensor, a capacitance type proximity sensor, amagnetic type proximity sensor, an infrared rays proximity sensor, andthe like. When the touchscreen is implemented as a capacitance type, theproximity sensor 541 can sense proximity of a pointer relative to thetouchscreen by changes of an electromagnetic field, which is responsiveto an approach of an object with conductivity. In this case, thetouchscreen (touch sensor) may also be categorized as a proximitysensor.

For clarity of the following description, the term “proximity touch”will often be referred to herein to denote the scenario in which apointer is positioned to be proximate to the touchscreen withoutcontacting the touchscreen. The term “contact touch” will often bereferred to herein to denote the scenario in which a pointer makesphysical contact with the touchscreen. For the position corresponding tothe proximity touch of the pointer relative to the touchscreen, suchposition will correspond to a position where the pointer isperpendicular to the touchscreen. The proximity sensor 541 may sense aproximity touch, and proximity touch patterns (for example, distance,direction, speed, time, position, moving status, and the like). Ingeneral, controller 580 processes data corresponding to proximitytouches and proximity touch patterns sensed by the proximity sensor 541,and causes output of visual information through the touchscreen. Inaddition, the controller 580 can control the mobile terminal 500 toexecute different operations or process different data according towhether a touch with respect to a point on the touchscreen is either aproximity touch or a contact touch.

A touch sensor can sense a touch applied to the touchscreen, such asdisplay unit 551, using any of a variety of touch methods. Examples ofsuch touch methods include a resistive type, a capacitive type, aninfrared type, and a magnetic field type, among others.

As one example, the touch sensor may be configured to convert changes ofpressure applied to a specific part of the display unit 551, or convertcapacitance occurring at a specific part of the display unit 551, intoelectric input signals. The touch sensor may also be configured to sensenot only a touched position and a touched area, but also touch pressureand/or touch capacitance. A touch object is generally used to apply atouch input to the touch sensor. Examples of typical touch objectsinclude a finger, a touch pen, a stylus pen, a pointer, and the like.

When a touch input is sensed by a touch sensor, corresponding signal(s)may be transmitted to a touch controller. The touch controller mayprocess the received signal(s), and then transmit corresponding data tothe controller 580. Accordingly, the controller 580 may sense whichregion of the display unit 551 has been touched. Here, the touchcontroller may be one of a component separate from the controller 580,the controller 580 itself, and combinations thereof.

In some embodiments, the controller 580 may execute the same ordifferent controls according to a type of a touch object that touchesthe touchscreen or a touch key provided in addition to the touchscreen.Whether to execute the same or different control according to the objectwhich provides a touch input may be decided based on a current operatingstate of the mobile terminal 500 or a currently executed applicationprogram, for example.

The touch sensor and the proximity sensor may be implementedindividually, or in combination, to sense various types of touches. Suchtouches include a short (or tap) touch, a long touch, a multi-touch, adrag touch, a flick touch, a pinch-in touch, a pinch-out touch, a swipetouch, a hovering touch, and the like.

If desired, an ultrasonic sensor may be implemented to recognizeposition information relating to a sensing target using ultrasonicwaves. The controller 580, for example, may calculate a position of awave generation source based on information sensed by an illuminationsensor and a plurality of ultrasonic sensors. Since light is much fasterthan ultrasonic waves, the time for which the light reaches the opticalsensor is much shorter than the time for which the ultrasonic wavereaches the ultrasonic sensor. The position of the wave generationsource may be calculated using this fact. For instance, the position ofthe wave generation source may be calculated using the time differencefrom the time that the ultrasonic wave reaches the sensor based on thelight as a reference signal.

The camera 521 is described as one component of the input unit 520 andtypically includes at least one of a camera sensor (CCD, CMOS etc.), aphoto sensor (or image sensors), and a laser sensor.

Implementing the camera 521 with a laser sensor may allow detection of atouch to a sensing target with respect to a 3D stereoscopic image. Thephoto sensor may be laminated on, or overlapped with, the displaydevice. The photo sensor may be configured to scan movement of thesensing target in proximity to the touchscreen. In more detail, thephoto sensor may include photo diodes and transistors at rows andcolumns to scan content received at the photo sensor using an electricalsignal which changes according to the quantity of applied light. Namely,the photo sensor may calculate the coordinates of the sensing targetaccording to variation of light to thus obtain position information ofthe sensing target.

The display unit 551 is generally configured to display (or output)information processed in the mobile terminal 500. For example, thedisplay unit 551 may display running screen information of anapplication program run on the mobile terminal 500, a user interface(UI) information in response to the running screen information, and/or agraphic user interface (GUI) information in response to the runningscreen information.

In some embodiments, the display unit 551 may be implemented as astereoscopic display unit for displaying stereoscopic images.

A typical stereoscopic display unit may employ a stereoscopic displayscheme such as a stereoscopic scheme (glass scheme), anauto-stereoscopic scheme (glassless scheme), a projection scheme(holographic scheme), or the like.

In general, a 3D stereoscopic image may include a left image (e.g., aleft eye image) and a right image (e.g., a right eye image). Accordingto how left and right images are combined into a 3D stereoscopic image,a 3D stereoscopic imaging method can be divided into a top-down methodin which left and right images are located up and down in a frame, anL-to-R (left-to-right or side by side) method in which left and rightimages are located left and right in a frame, a checker board method inwhich fragments of left and right images are located in a tile form, aninterlaced method in which left and right images are alternately locatedby columns or rows, and a time sequential (or frame by frame) method inwhich left and right images are alternately displayed on a time basis.

Also, as for a 3D thumbnail image, a left image thumbnail and a rightimage thumbnail can be generated from a left image and a right image ofan original image frame, respectively, and then combined to generate asingle 3D thumbnail image. In general, the term “thumbnail” may be usedto refer to a reduced image or a reduced still image. A generated leftimage thumbnail and right image thumbnail may be displayed with ahorizontal distance difference there between by a depth corresponding tothe disparity between the left image and the right image on the screen,thereby providing a stereoscopic space sense.

A left image and a right image required for implementing a 3Dstereoscopic image may be displayed on the stereoscopic display unitusing a stereoscopic processing unit. The stereoscopic processing unitcan receive the 3D image and extract the left image and the right image,or can receive the 2D image and change it into a left image and a rightimage.

The sound output module 552 is generally configured to output audiodata. Such audio data may be obtained from any of a number of differentsources, such that the audio data may be received from the radiocommunication unit 510 or may have been stored in the memory 560. Theaudio data may be output during modes such as a call signal receptionmode, a call mode, a record mode, a voice recognition mode, a broadcastreception mode, and the like. The sound output module 552 can provideaudible output related to a particular function (e.g., a call signalreception sound, a message reception sound, etc.) performed by themobile terminal 500. The sound output module 552 may also include areceiver, a speaker, a buzzer, and/or the like.

A haptic module 554 can be configured to generate various tactileeffects that can be sensed by a user. A typical example of a tactileeffect generated by the haptic module 554 is vibration. The strength,pattern and the like of the vibration generated by the haptic module 554can be controlled by a user's selection or a setting of the controller.For example, the haptic module 554 may combine and output differentvibrations together or may output different vibrations in a sequentialmanner.

As well as the vibration, the haptic module 554 can generate variousother tactile effects, including an effect by stimulation such as a pinarrangement moving vertically to a contacted skin surface, a spray orsuction force of air through a jet orifice or a suction opening, a touchto a skin surface, a contact of an electrode, an electrostatic force, aneffect by reproducing the cold/warm sense using an endothermic orexothermic element, and the like.

The haptic module 554 can also be implemented to allow the user to feela tactile effect through a muscle sensation such as the user's fingersor arm, as well as to transfer the tactile effect through directcontact. Two or more haptic modules 554 may be provided according to aconfiguration type of the mobile terminal 500.

An optical output unit 553 can output a signal for indicating an eventoccurrence using light of a light source. Examples of events occurringin the mobile terminal 500 may include a message reception, a callsignal reception, a missed call, an alarm, a schedule notification, anemail reception, an information reception through an application, andthe like.

A signal output by the optical output unit 553 may be implemented insuch a manner that the mobile terminal emits monochromatic light orlight with a plurality of colors. The signal output may be terminated asthe mobile terminal senses that a user has checked the event forexample.

The interface unit 570 serves as an interface for all external devicesconnected to the mobile terminal 500. For example, the interface unit570 can receive data transmitted from an external device, receive powerto transfer to elements and components within the mobile terminal 500,or transmit internal data of the mobile terminal 500 to such externaldevice. The interface unit 570 may include wired/wireless headset ports,external power supply ports, wired/wireless data ports, memory cardports, ports for connecting a device having an identification module,audio input/output (I/O) ports, video I/O ports, earphone ports, and/orthe like.

The identification module may include a chip configured to store variousinformation for authenticating authority in using the mobile terminal500 and may include a user identity module (UIM), a subscriber identitymodule (SIM), a universal subscriber identity module (USIM), and thelike. In addition, the device having the identification module (alsoreferred to herein as an “identifying device”) may be manufactured inthe form of a smart card. Hence, the identifying device can be connectedwith the terminal 500 via the interface unit 570.

When the mobile terminal 500 is connected with an external cradle, theinterface unit 570 can serve as a passage to allow power from the cradleto be supplied to the mobile terminal 500 or may serve as a passage toallow various command signals inputted by the user from the cradle to betransferred to the mobile terminal there through. Various commandsignals or the power inputted from the cradle may operate as signals forrecognizing that the mobile terminal is properly mounted on the cradle.

The memory 560 can store programs to support operations of thecontroller 580 and may temporarily store input/output data (for example,phonebook, messages, still images, videos, etc.). The memory 560 maystore data related to various patterns of vibrations and audio that areoutputted in response to touch inputs to the touchscreen.

The memory 560 may include at least one of types of storage mediumsincluding a flash memory, a hard disk, a solid state disk, a silicondisk, a multimedia card micro type, a card-type memory (e.g., SD memory,XD memory, etc.), a Random Access Memory (RAM), a Static Random AccessMemory (SRAM), a Read-Only Memory (ROM), an Electrically ErasableProgrammable Read-Only Memory (EEPROM), a Programmable Read-Only memory(PROM), a magnetic memory, a magnetic disk, an optical disk, and thelike. The mobile terminal 500 may also operate in relation to a webstorage device that performs the storage function of the memory 560 onInternet.

As mentioned in the foregoing description, the controller 580 controlsoperations related to the application programs and may typicallycontrols overall operations of the mobile terminal 500. For example, thecontroller 580 may set or release a lock state for restricting a userfrom inputting a control command with respect to applications if astatus of the mobile terminal meets a preset condition.

The controller 580 may perform the controlling and processing associatedwith voice calls, data communications, video calls, and the like, or mayperform pattern recognition processing to recognize a handwriting inputor a picture drawing input performed on the touchscreen as characters orimages, respectively. In addition, the controller 580 can control one ora combination of those components mentioned in the foregoing descriptionin order to implement various embodiments mentioned in the followingdescription.

The power supply unit 590 receives an external or internal power underthe control of the controller 580 and then supplies the power requiredfor operating the respective elements and components. The power supplyunit 590 may include a battery, which is typically rechargeable or bedetachably coupled to the terminal body for charging.

The power supply unit 590 may include a connection port. The connectionport may be configured as one example of the interface unit 570 to whichan external charger for supplying power to recharge the battery iselectrically connected.

As another example, the power supply unit 590 may be configured torecharge the battery in a wireless manner without use of the connectionport. In doing so, the power supply unit 590 can receive power,transferred from an external wireless power transmitter, using at leastone of an inductive coupling method which is based on magnetic inductionor a magnetic resonance coupling method which is based onelectromagnetic resonance.

Meanwhile, various embodiments described herein may be implemented in acomputer-readable medium, a machine-readable medium, or similar mediumusing, for example, software, hardware, or any combination thereof.

In the foregoing description, the smart controlling device 110 isexplained as a separate configuration in the digital system and adigital TV and a mobile terminal are respectively illustrated in FIG.4A, FIG. 4B and FIG. 5 as an example of a terminal performing datacommunication with the smart controlling device. However, the smartcontrolling device 110 according to the present invention can also beimplemented by a single configuration such as a type capable of beingembedded or combined with the digital TV or the mobile terminal shown inFIG. 4A, 4B or FIG. 5. Meanwhile, the smart controlling device 110 canbe referred to by various terms such as an audio device, a video device,and the like depending on an implementation type of the smartcontrolling device.

FIG. 6A is a conceptual diagram for an execution state of a voicerecognition function and FIG. 6B is a flowchart for a method ofexecuting a voice recognition function according to embodiments of thepresent disclosure.

A voice recognition function corresponds to converting an audio signalreceived through an audio sensor such as a microphone to text. The voicerecognition function may further include performing a specific operationbased on the information converted into text. In particular, the voicerecognition function corresponds to a function for determining whetheror not a received voice corresponds to a specific word or a function fordetecting a voice command.

The smart controlling device 110 according to the present invention canstore voice recognition application and information related to the voicerecognition function in the memory 240. The voice recognitionapplication can perform voice recognition via a database installed inthe smart controlling device or a database installed in acommunication-connected server.

A voice recognition function may have a plurality of execution states.Specifically, the controller 220 can set an execution state of the voicerecognition function to either a voice recognition standby state 610waiting for an input of a command trigger or a recognition state 620performing voice recognition by analyzing voice to start the voicerecognition function.

The voice recognition standby state 610 can include a listening state611 for sensing a voice of a volume equal to or greater than aprescribed volume and a keyword detecting state 612 for detecting aspecific word.

The controller 220 can sense reception of a voice of a volume equal toor greater than a prescribed volume in the listening state 611. Thecontroller may 220 only detect the voice of the volume equal to orgreater than the prescribed volume in the listening state 611 and maynot perform voice recognition functions.

If a voice input equal to or greater than the prescribed volume isreceived, the controller 220 can switch the execution state of the voicerecognition function to the keyword detecting state 612 from thelistening state 611.

The controller 220 can detect whether or not a specific word wasreceived in the keyword detecting state 612. The specific wordcorresponds to a command trigger for starting the voice recognitionfunction. It may possible to configure a different signal for thecommand trigger depending on a smart controlling device 110 or anapplication providing the voice recognition function.

If a voice input corresponding to a specific word is received in thekeyword detecting state 612, the controller 220 can switch to therecognition state 620. For example, if a predetermined specific wordsuch as “Alice” is received as a voice input in the keyword detectingstate 612, the controller 220 may detect the specific word and switchthe execution state of the voice recognition function to the recognitionstate 620.

On the contrary, if a voice corresponding to a specific word is notreceived in the keyword detecting state 612, the controller 220 canswitch the execution state of the voice recognition function back to thelistening state 611. For example, if a voice is not received for morethan a prescribed time in the keyword detecting state 612, thecontroller 220 can switch the execution state of the voice recognitionfunction to the listening state 611. Or, if a voice received in thekeyword detecting state 612 does not correspond to a specific word, thecontroller 220 can switch the execution state of the voice recognitionfunction back to the listening state 611.

In the following, a method of executing a voice recognition function inthe recognition state is explained in more detail with reference to FIG.6B.

Referring to FIG. 6B, if the controller 220 receives a command triggerin the keyword detecting state 612 [S610], the controller 220 can startthe voice recognition function. Subsequently, the controller 220 canreceive a voice input in the recognition state 620 [S620].

If a voice input is received in the recognition state 620, thecontroller 220 can analyze the voice based on various predeterminedalgorithms for voice and speech analysis which is well known in the art.Since the contents of the algorithm are well known to those skilled inthe art, explanation on the contents is omitted in the presentspecification.

The controller 220 can detect at least one function corresponding to thevoice input based on a result of the analysis. In particular, thecontroller 220 may detect one or a plurality of functions correspondingto the voice input.

The controller 220 can determine whether or not one or more functionscorresponding to the voice input is detected [S630]. If a functioncorresponding to the voice input is detected, the controller 220 canexecute a first function corresponding to the voice input [S642].

The first function may correspond to either a function configured as abasic function or a function of high priority. When a plurality offunctions corresponds to a single voice input, the basic function may bethe first to be executed among the plurality of the functions. Thepriority can be determined according to various factors such asalphabetical order, an execution frequency, a use pattern, or a musicsource chart ranking. For example, if a function for playing songs isdetected, priority can be determined by a descending order of a playbackfrequency of each song.

For example, the voice input may include the term “Girl's generation”and the controller may determine that the user is commanding playback ofa song corresponding to the musical group Girls Generation. The voiceinput may also include the name of a particular song by the musicalgroup. In this case, the controller 220 can execute the ‘Girlsgeneration music playing function’ as a first function.

In the foregoing description, when a plurality of functionscorresponding to a voice input is detected, a first function isexecuted. However, it may be able to output notification information tomake a user input a voice input again without executing the firstfunction. For example, the controller 220 may output a query to requesta more specific voice input, such as “A plurality of music sources isdetected. Which would you like to play?”.

Meanwhile, if a single function is detected corresponding to a voiceinput, the controller 220 can immediately execute the single functioncorresponding to the voice input [S641]. For example, if the controller220 receives a voice input such as ‘play track 1’, the controller candetermine and play a music source stored in the first track. If aspecific operation is performed in the recognition state 620 based on avoice input, the controller 220 can switch the execution state of thevoice recognition function back to the standby state 610. By doing so,the controller 220 can selectively analyze voice recognition-requiredvoices only without analyzing all voices received via a microphone,thereby preventing unnecessary power consumption.

In order to input a voice input, a user may speak a a specific commandtrigger followed by a voice command. However, when the user wishes toinput multiple voice commands, the user may be required to speak thecommand trigger for each command multiple times, which may beinconvenient and lead to inaccurate voice input recognition.

FIG. 7 is a flowchart depicting a method of processing data of a smartcontrolling device according to an embodiment of the present invention.

In the following description, we assume that the smart controllingdevice 110 is in an execution or an executable state.

First of all, the smart controlling device 110 may wait for a commandtrigger [S710] and determines whether or not the command trigger isreceived [S720].

If the command trigger is received, the smart controlling deviceswitches to a general command standby state for receiving command data[S730].

As mentioned earlier in the step S730, if the command data is receivedafter the smart controlling device is switched to the general commandstandby state, the smart controlling device processes the command datato generate a result data [S740].

And, the smart controlling device generates a response from the resultdata and plays the response [S750].

FIG. 8 is a flowchart depicting a method of processing data of a smartcontrolling device according to one embodiment of the present invention.

If the command trigger is received in the standby state [S802] asdiscussed, the smart controlling device 110 is switched to a commanddata received state and detects a location of a speaker (i.e., user) whohas spoken the trigger data [S804].

If the location of the user is detected, the smart controlling device110 detects whether or not a display device exists within apredetermined range on the basis of the detected user location [S806].In some cases, the user location, which becomes a criterion fordetecting whether or not the display device exists, can be based on thesmart controlling device 110. In other words, detecting whether or notthe display device exists may correspond to a location of the smartcontrolling device 110 rather than the user.

Meanwhile, the display device may correspond to any electronic devicesequipped with a display including the digital TV shown in FIG. 4A, 4Band the mobile terminal shown in FIG. 5. As a result of detecting thedisplay device in the step S806, if a plurality of display devices isdetected, the smart controlling device 110 can select one from among aplurality of the display devices. In this case, the smart controllingdevice can select a display device on the basis of a current position ofa user (or the smart controlling device 110). In particular, the smartcontrolling device can select a display device closest from the user (orthe smart controlling device 110). Or, a display device including adisplay of which a current display state corresponds to a standby stateor power is turned on can be selected from among a plurality of thedisplay devices.

If there are a plurality of display devices including a display that thecurrent display state correspond to the standby state or the power isturned on, the smart controlling device can select a display deviceclosest from the smart controlling device from among a plurality of thedisplay devices on the basis of the current position of the user (or thesmart controlling device 110). Or, it may consider not only the currentposition of the user (or the smart controlling device 110) but also adirection that the user is facing. For example, when a user is lookingat a direction corresponding to 12 o'clock, or is moving toward thatdirection, if a display device positioned at a direction correspondingto 6'oclock is selected, it would be necessary for the user to turnaround or move in the opposite direction to watch the display of theselected display device, which is undesired. The direction of the usermay become another factor for selecting a display.

In the foregoing description, for clarity, although it is explained as asingle display is selected from among a plurality of the detecteddisplay devices, the present invention may not be limited to this. Insome embodiments, the smart controlling device 110 can select aplurality of display devices from among a plurality of the displaydevices detected in S806. In this case, although a plurality of thedisplay devices may be selected, it may be able to put priority on oneor more of the selected display devices with reference to theaforementioned criteria and control each of the plurality of theselected display devices accordingly based on priority. By doing so, itmay be able to flexibly cope with various situations such as an error, anetwork problem, and the like in a relation with a specific displaydevice.

In the step S806, if a display device adjacent to the user (or smartcontrolling device 110) is detected and selected on the basis of acurrent position, the smart controlling device 110 may determine a powerstatus of the selected display device [S808]. In this case, the powerstatus of the display device may correspond to a power-off status, astand-by status, or a power-on status.

The smart controlling device 110 determines whether or not a displaypower status of the display device corresponds to the power-on status[S801]. If the display power status is not the power-on status, thesmart controlling device 110 transmits a control command to change thedisplay power status to a stand-by mode or the stand-by status [S812].In this case, as a result of the step S810, if the display power statusis not the power-on status but already in the stand-by status, the stepS812 can be omitted.

If necessary, an operation by the display device for performing mutualdata communication with the smart controlling device 110 can be includedas well. For example, the operation can include an operation ofestablishing a Bluetooth interface, WiFi interface, and the like, anoperation of turning on power, authentication, registration, and thelike.

Subsequently, if the display power status of the display device isswitched to the stand-by status, the smart controlling device 110analyzes a voice input, i.e., command data, which is received after thestep S802. After the command data is analyzed, the smart controllingdevice determines whether or not a response data corresponds to a signalrequired to be outputted via the display of the selected display device[S814]. In other words, the smart controlling device 110 determineswhether or not the command data requires an audio response to be outputby the smart controlling device, or whether a non-audio response isrequired to be output by a different device, such as output of an imageformat, a text format, or the like.

As a result of the step S814, if the response data is determined torequire output via the display, the smart controlling device 110transmits a control command for turning on power of the display to theselected display device to switch the display power to the power-onstatus from the stand-by status [S816].

The smart controlling device 110 transmits response data to be outputtedvia the turned-on selected display device and a signal for controllingthe response data to be outputted to the display device [S818]. Thedisplay device outputs the response data via the display according tothe control of the smart controlling device 110 [S822]. Meanwhile, inrelation to the output of the response data, if other data was beingoutputted via the display or an application was being executed at thedisplay device, the smart controlling device 110 or the display devicestops (or switches to the background) the previously displayed orexecuted data or application and causes output of the response data. Theresponse data may also be output in various forms, such as a fullscreen, a pop-up window, split screen, overlay on a prescribed area of ascreen, and the like.

When the display device can output audio data of the responses data inaddition to displaying the response data. Additionally, guide data(indicating start of display, a type of display, identification of adisplay device, or the like), which is related to the output of theresponse data, can be outputted via an audio output device such as aspeaker of the smart controlling device 110 or the display device usingan output format (e.g., audio output format) different from thedisplayed response data.

Referring back to S810, if power of the selected display device isalready in the turned-on status, as mentioned earlier in the step S814,the smart controlling device analyzes a command data, which is receivedafter the command trigger is received, and determines whether or not itis necessary to output a response data via the display of the displaydevice [S818]. In this case, as shown in S820, the response data isoutputted on the display. Yet, as a result of the step S814 or S818, ifit is not necessary to output the response data via the display of thedisplay device, in other words, if it is sufficient to output theresponse data using an audio format only, the response data is outputtedin an audio format via an audio output device (signal outputting unit)of the smart controlling device 110 [S824] and the switching of thedisplay stand-by mode of the selected display device is cancelled[S826]. In another embodiment, if an audio output device of a previouslyselected display device is being used rather than the audio outputdevice of the smart controlling device 110, the switching of thestand-by mode may not be cancelled. As mentioned in the foregoingdescription, it may be able to control the turn-on status or the currentstand-by status to be maintained and use the audio output device foroutputting audio of the response data.

FIG. 9 is a diagram depicting a method of controlling a display deviceof a smart controlling device according to one embodiment of the presentinvention.

For example, FIGS. 9a and 9b show a situation where a plurality ofdisplay devices is near the smart controlling device and FIG. 9c shows asituation where only a single display device is near the smartcontrolling device.

First of all, referring to FIG. 9a , while a user is watching a firstdisplay device 910, the user may input a command trigger to the smartcontrolling device 110. In this case, the first display device 910 maybe located outside of a predetermined distance range from the smartcontrolling device 110 or the user.

If the command trigger is received at the smart controlling device andit is determined that the first display device 910 is located outside ofthe predetermined or threshold distance range, the smart controllingdevice 110 may instead detect a second display device 920 located withinthe predetermined distance range. In some cases, the first displaydevice may not be detectable by the smart controlling device due to thedistance, or the smart controlling device may detect the first displaydevice and its relative distance, and determine that it is outside ofthe predetermined distance range. The smart controlling device 110 maydetermine a display power status of the detected second display device920. If the display power status is off, the smart controlling devicetransmits a first control command to the second display device to switchthe display power status of the second display device to a stand-bymode.

The second display device switches the display power status to thestand-by mode from the off status according to the first control commandand maintains the stand-by mode.

As shown in FIG. 9b , the smart controlling device 110 analyzes commanddata which is received after the command trigger. The smart controllingdevice 110 determines a format of a response output based on a result ofanalysis to determine a format of a response to be transmitted inresponse to the command data. The response data may be and output of anaudio format, a text format, an image format, a video format, and thelike. If the response output format is not the audio format, the smartcontrolling device 110 transmits a second control command to the seconddisplay device to switch the display power status of the second displaydevice to the power-on status from the stand-by status. At the same timeor after display power of the second display device is turned on, thesmart controlling device 110 transmits response data to the seconddisplay device 920 to be output in response to the command data andcontrols the response data to be outputted via the display.

If the response data is outputted via the second display device 920,since the response for the command data is completed, the smartcontrolling device 110 can further control the display power status ofthe second display device 920. For example, when the response data hasbeen outputted via the second display device 920 or the playback of theresponse data is terminated, if an additional command data (or commandtrigger) of the user is not received within a prescribed time, the smartcontrolling device 110 can control the display power status of thesecond display device 920 to be switched back to a previous status,i.e., the stand-by status or the power-off status. If the display powerstatus of the second display device 920 is switched to the stand-bystatus, the smart controlling device 110 can transmit an additionalcontrol command to the second display device 920 to control the displayto be power-on or power-off according to whether or not an additionalvoice command signal is received within a prescribed time.

Even where the first display device 910 is located within thepredetermined range, the smart controlling device may select the seconddisplay device 920 so that the currently displayed content is notinterrupted or blocked. Further, the smart controlling device may beable to select another display device more appropriate for a display ofa response data based on a closer distance from the user or the smartcontrolling device, a gaze direction of the user, or a moving directionof the user.

FIG. 9c shows a situation where there is no display device outputtingcontent and only a single display device 930 is present within thepredetermined threshold range. In this case, as mentioned in theforegoing description, the smart controlling device 110 detects thedisplay device 930 according to a command trigger and controls thedisplay device 930 to be switched to a stand-by mode according to adisplay power status of the detected display device 930 and controlsdata to be displayed at the display device 930 according to analysis ofcommand data.

If at least one display device is not detected within a predeterminedrange (e.g., first range) on the basis of a user (speaker), the smartcontrolling device 110 can perform an additional operation depending ona configuration. For example, the additional operation may correspond torepeatedly attempting to detect a display device up to a maximum numberof attempts or continuously attempting to detect without a maximumnumber of attempts until at least one display device is detected withinthe first range. In another embodiment, the first range may be expandedto a greater second range or an even greater third range to detect adisplay device.

Even where at least one display device is detected within the range, theadditional operation may also include where a display power status ofthe display device does not respond to a stand-by mode control or aturn-on control or a case that it is difficult to execute the stand-bymode control or the power-on control signal. In some cases, if a validdisplay device is not detected within the predetermined range or anexpanded range, the smart controlling device may be configured to outputthe response data at the smart controlling device, such as via thedisplay or an audio speaker, or control a different type of device tooutput the response data. This may include changing or converting theoutput format of the response data to a format which can be output bythe smart controlling device or by the different type of device, such asconverting to the audio output format. If the smart controlling device110 is equipped with a display, the output format can also be outputtedvia the display of the smart controlling device. In this case, anindication may be output that the format has been modified, such as anaudio indicator.

FIG. 10 is a flowchart depicting a method of outputting data of a smartcontrolling device according to another embodiment of the presentinvention.

In particular, FIG. 10 shows a method of delivering a command triggerand response data of a command data, i.e., feedback.

The smart controlling device 110 receives a prescribed signal input in acommand trigger stand-by state [S1002]. In this case, the prescribedsignal can be received in various types including a voice of a speaker(user), a text input by the user, a motion gesture of the user, and thelike. For clarity, assume that the prescribed signal is received as avoice input.

If the prescribed signal is received, the smart controlling device 110determines whether or not the inputted signal corresponds to a commandtrigger [S1004].

If the inputted signal corresponds to a command trigger, the smartcontrolling device 110 is switched to a general command stand-by stateto receive a command data [S1012] or performs an operation according tothe command data. And part of or all of S1006, S1008, and S1010, areperformed simultaneously with S1012.

In this case, the operation detects a location of a user who has inputthe prescribed signal [S1006] and detects a neighboring display devicesuch as a TV, a mobile terminal, a tablet, a refrigerator, or the like,including a display based on the detected location of the user [S1008].After the step S1008 is performed, the smart controlling device 110determines a display status or a display power status of the detecteddisplay device [S1010].

After the step S1010 is performed, the smart controlling device 110receives and analyzes a command data of the user which is inputted afterthe command trigger to determine an output format of a response. Thesmart controlling device 110 determines whether or not the output formatfor the response data corresponds to an audio output format. Inparticular, the smart controlling device 110 determines whether adisplay is required for the response data [S1020].

If the output format of the response data is an audio output for which adisplay is not necessary, the smart controlling device 110 cause outputof the audio response data [S1022] when the feedback is not performed.In particular, the smart controlling device 110 may transmit a controlcommand to cause a neighboring display device to be turned off andoutputs the response data in the audio format. In this case, the audioformat can be directly outputted by the smart controlling device 110 orcan be outputted by an external audio output device which is selected bya scheme similar to the operation [S1010]. For example, the smartcontrolling device may detect an audio output device located within apredetermined range and determine whether or not the detected audiooutput device is able to output audio. Subsequently, the smartcontrolling device may cause the detected audio output device to outputresponse data after being switched to a stand-by mode or turn-on state.The smart controller may similarly cause the detected audio outputdevice to be switched back to a turn-off state after outputting theresponse data.

Meanwhile, referring back to S1020, if the output format of the responsedata requires output of the response data on a display rather than anaudio output, the smart controlling device 110 turns on the power of thedisplay of the neighboring display device, which is selected accordingto the operation [S1008], and transmits a control command to cause thedisplay device to output the response data [S1024].

Besides, in case of the step S1012, a command data is received andanalyzed in a general command stand-by state to generate a commandresult, i.e., a response data [S1014].

When the generated response data is output according to a command spokenby the user (i.e., command data), the smart controlling device 110determines whether to use the display [S1016]. If it is necessary to usethe display for the response data output, as mentioned in the foregoingdescription, the smart controlling device 110 performs the step [S1024]using the neighboring display device. In this case, if the operation[S1010] according to the present invention is not performed yet, thesmart controlling device 110 may perform the steps S1006 to S1010.

However, after the step S1016 is performed, if it is not necessary touse the display for the feedback, the smart controlling device outputsthe generated response data in an audio output form [S1018].

FIGS. 11 to 14 are diagrams depicting various examples of a smartcontrolling device according an embodiment of the present invention.

Referring to FIG. 11a , if a voice input including a command trigger anda command signal, such as “Alice, what is a Wednesday/Thursday soapopera on TV tonight?” is received, the smart controlling device 110analyzes the voice input. As a result, if it is determined that anoutput format of response data does not require a display of a displaydevice, as shown in FIG. 11b , the smart controlling device can directlyoutput audio feedback in response to the command data, for example, itmay output the answer “Shin Saimdang starts at 9:50 on SBC” as audioonly.

FIGS. 12 to 14 show examples of response data output via a displayrather than the audio output format. In this case, a digital TV, arefrigerator equipped with a display, and a tablet PC are shown in FIG.12, FIG. 13, and FIG. 14, respectively, as a display device.

Referring to FIG. 12a , the smart controlling device 110 receives avoice input such as “Alice, show me today's TV program listings” whichincludes a command trigger and command data. The smart controllingdevice 110 may detect a location of the user based on the voice inputand detect a neighboring display device based on the detected locationof the user. The smart controlling device 110 determines a display powerstatus of the detected neighboring display device. If the display powerstatus of the neighboring display device is determined as an off status,the smart controlling device transmits a control command to theneighboring display device to change the display power status to astand-by status.

While the detecting procedures and transmission of the control commandare performed, or after the detecting procedures and the transmission ofthe control command are performed, the smart controlling device 110analyzes an input signal, i.e., a command data, which is received afteror with the command trigger. The smart controlling device 110 determineswhether to use the neighboring display device (i.e., output format of aresponse data) according to a result of the analysis. If it isdetermined that the neighboring display device is not to be used, thesmart controlling device 110 can transmit a control command to theneighboring display device to make power of the neighboring displaydevice to be turned off In this case, the smart controlling device 110can transmit the turn-off control command to all detected displaydevices. In other embodiments, the smart controlling device 110 may nottransmit the turn-off control command to a display device in which anapplication is currently executed or content is currently beingdisplayed to save power consumption and to prevent interrupting currentusage.

On the contrary, if it is determined to use the neighboring displaydevice, in one embodiment, the smart controlling device 110 can controlTV program listings 1220 to be outputted on a screen of the neighboringdisplay device 1210. In this case, for example, if the neighboringdisplay device stores or has access to TV program listings data, thesmart controlling device 110 can transmit a control command fordisplaying the TV program listings data only. In other embodiments, thesmart controlling device 110 may receive the TV program listings datafrom an external server and provide the TV program listings data to theneighboring display device, or cause the TV program listings data to betransmitted directly to the display device.

As shown in FIG. 12b , when the TV program listings is outputted via thedisplay device 1210 as a response data, an audio indicator may be outputto explain the information being output, for example, “These are theprogram listings from this time forward.” Since it may be difficult fora user to know which display device is outputting the response data, theaudio indicator may be output from the display device before the TVprogram listings is output on the display device 1210 to make the useraware of the display device 1210 that is going to output the responsedata.

FIG. 13 shows an example of using a refrigerator equipped with adisplay. For example, as mentioned in the foregoing description,selection of a neighboring display device may not be based only on alocation of the user or a direction of the user, but the selection mayalso be based on an attribute of the command data or an attribute of aresponse data, and the like.

Referring to FIG. 13a for example, if a voice input such as “Alice, giveme a recipe for braised spicy chicken” is received from a user, thesmart controlling device 110 selects a neighboring display device basedon the command trigger and the command data, and the type of responsedata required (recipe for braised spicy chicken-preparation material,cooking method/process, etc.). In this case, the response data caninclude not only text of a recipe, but also other data including animage, a video, and the like, related to the recipe. Since the commanddata or the response data is determined to be related to food or cookingrecipes, a display device of a refrigerator may be selected. This may bethe case even where other display devices are located closer to thesmart controlling device. The smart controlling device may accordinglytake all factors of each display device into consideration andprioritize them, including distance, purpose, type, functionality, andthe like. As mentioned in the foregoing description, when the responsedata is outputted via the display device, it may also be able to outputaudio indicators (“The recipe is displayed on the refrigerator”).

In the foregoing description, the command trigger can be recognized as astarting point or initiating term for detecting a nearby display deviceand controlling a display power status of the display device. Forexample, when a signal is inputted to the smart controlling device 110,the smart controlling device can be configured to recognize the signalas a valid input for the smart controlling device 110 only when acommand trigger is included in the signal. In other cases, when a userinputs voice commands to the smart controlling device 110, the user mayinput a command trigger in addition to the command data at the sametime, or within a short time thereafter, instead of separately sendingthe command trigger and the command data. In this case, if an inputsignal including a command trigger is received from the user, the smartcontrolling device 110 determines the input signal as a valid user inputand detects a location of the user.

When a display device is selected based on the location of the user, thedisplay device can also be selected by analyzing a command data includedin the input signal. In other words, as shown in FIG. 13, if anattribute of a command data is recognized as related to food or cooking,the smart controlling device is able to prevent an error and increaseefficiency in detecting display devices. This is because, if anintention of a user relates to food or cooking, it may be preferable touse a display device related to food or the kitchen instead of receivingthe response data on a TV in another area. In this case, the smartcontrolling device 110 may change the predetermined range for detectingdisplay devices or expand the predetermined range. Or, it may base theselection of display devices on factors other than distance.

Referring to FIG. 14a , if an input signal such as “Alice, show me LOTmusic video for refreshing” is received from a user, the smartcontrolling device 110 selects a neighboring display device (tablet PC1410) according to a prescribed criterion and controls a music videodata to be outputted via the selected display device as response data.In this case, the tablet PC 1410 executes an application, i.e., a videoplayer, corresponding to the response data according to the control ofthe smart controlling device 110 and outputs the response data(including lyrics in some embodiments) via the executed music player.

In FIG. 14, the smart controlling device may select the display devicebased on a display power status, a resource, and the like, in additionto the criterion for selecting the display device. For example, when aformat of the input signal received from the user and/or a format of theresponse signal corresponds to a specific format, the smart controllingdevice will attempt to select a display device capable of receiving,processing, or playing back the specific format. Hence, when the smartcontrolling device 110 detects or selects a display device or aneighboring display device, the smart controlling device may refer todata capabilities of a resource in advance. When provided with more thanone resource with sufficient capabilities, the smart controlling devicemay further consider other factors, including a usage pattern learnedfrom a user, a resolution of a data, and the like, as well as theaforementioned distance, directivity, attributes, and the like.

FIG. 15 is a flowchart depicting a method of outputting data of a smartcontrolling device according to a further embodiment of the presentinvention.

In particular, FIG. 15 relates to providing image information. Whenpower of the smart controlling device 110 is initially turned on, it maycheck basic information on a display device in advance and use the basicinformation.

The smart controlling device 110 receives a prescribed signal in acommand trigger stand-by state [S1502]. In this case, the prescribedsignal can be received in various types including a voice, a text, amotion gesture, and the like. For the purposes of the example, we assumethat the prescribed signal corresponds to a voice signal.

If the prescribed signal is received, the smart controlling device 110determines whether or not the inputted signal corresponds to commandtrigger or whether or not the inputted signal is included in a commandtrigger [S1504].

If the inputted signal corresponds to a command trigger or is includedin the command trigger, the smart controlling device 110 performsoperations according to the present invention.

In this case, the operations include detecting a location of a user whohas input the prescribed signal and detect a neighboring display devicesuch as a TV, a mobile terminal, a tablet, a refrigerator including adisplay device, and the like, based on the detected location of theuser. In particular, the smart controlling device detects a displaydevice capable of being connected with the smart controlling device.[S1506]. The smart controlling device 110 may further check imageinformation of the detected display device [S1508]. S1506 and S1510 areperformed simultaneously. Specifically, the image information is theimage information playing on the screen of the detected display device.If the detected display device displays movie program, the imageinformation may be movie program image information. So the smartcontrolling device 110 may check what is playing in the detected displaydevice and obtain movie program image information from the detecteddisplay device.

The smart controlling device 110 may receive and analyzes command datawhile in a general command stand-by state [S1510] and generates acommand result, i.e., a response data [S1512]. In this example, thegenerated response data may correspond to image information to be outputin response to the command data.

The smart controlling device 110 determines whether or not there is a adisplay device at which image information corresponding to the responsedata is to be output [S1514]. If there exists a display device at whichthe image information can be output, the smart controlling devicedetermines a power status of the display device [S1518]. The smartcontrolling device causes the power of the display device to be switchedto a stand-by state [S1520] when the smart controlling device determinethat it is not power status but standby state and causes the imageinformation to be output via the display device [S1522].

If no display device is detected at which the image information can beoutput, the smart controlling device may output the response data in anaudio output format [S1516]. The smart controlling device may beconfigured to convert the response data to audio output information,such as a description of the response data using image recognition,facial recognition, and the like.

FIG. 16 is an illustration of another embodiment of a smart controllingdevice according to the present invention.

Referring to FIG. 16a , if a signal such as “Alice, send a TV image to atablet” is received from a user, the smart controlling device 110detects both a TV 1610 and a tablet 1630.

As shown in FIG. 16b , the smart controlling device 110 transmits acontrol command to the TV 1610 to generate a screen shot image of ascreen 1620 of the TV 1610 to be transmitted to the detected tablet PC1630. In other embodiments, the TV may transmit currently displayedinformation such that the TV screen is continuously mirrored on thetablet PC. If necessary, control commands for transmitting the screenshot image to the tablet PC and controlling playback of the transmittedimage can be separately performed. If the screen shot image 1620 isreceived from the TV 1610, the tablet PC 1630 controls an appropriateapplication (e.g., video or image player application) to be executedaccording to the screen shot image for display or playback of the screenshot image 1620 using the executed video player application 1640.

FIG. 17 is a flowchart depicting a method of outputting data of a smartcontrolling device according to another embodiment of the presentinvention.

FIG. 17 illustrates a case of obtaining image information from adifferent device.

The smart controlling device 110 receives a prescribed signal in acommand trigger stand-by state [S1702]. In this case, the prescribedsignal can be received in various types including a voice, a text, amotion gesture, and the like. For this example, we assume that theprescribed signal corresponds to a voice signal.

If the prescribed signal is received, the smart controlling device 110determines whether or not the inputted signal corresponds to a commandtrigger or whether or not the inputted signal is included in a commandtrigger [S1704].

If the inputted signal corresponds to the command trigger or is includedin the command trigger, the smart controlling device 110 performsoperations according to the present invention. And, S1706 and S1710 areperformed simultaneously.

In this case, the operations includes detecting a location of a user whohas transmitted the prescribed signal and detecting a neighboringdisplay device such as a TV, a mobile terminal, a tablet, a refrigeratorincluding a display device, and the like, based on the detected locationof the user. In particular, the smart controlling device detects adisplay device capable of being connected with the smart controllingdevice. [S1706]. The smart controlling device 110 further checks imageinformation of the detected display device [S1708]. Specifically, theimage information is the image information playing on the screen of thedetected display device. If the detected display device displays Newsprogram, the image information may be News program image information. Sothe smart controlling device 110 may check what is playing in thedetected display device and obtain News program image information fromthe detected display device.

The smart controlling device 110 receives and analyzes a command data ina general command stand-by state [S1710] and generates a command result,i.e., a response data [S1712].

The smart controlling device 110 determines whether or not there is aneighboring display device to which the image information is to beobtained [S1714]. A display device can be a table PC. When the smartcontrolling device feedbacks the response data, if there exists thelocation and/or the display to which the image information is to beobtained, the smart controlling device determines a power status of thedisplay device [S1718]. The smart controlling device controls displaypower to be switched to a stand-by state [S1720] when the controllingdevice determines that it is not a power status but a stand-by state[S1718] and shares an image via the display device [S1722].

If there is no location and/or a display at which the image informationis to be shared, the smart controlling device may generate response andplay. Specifically, the smart controlling device may output the responsedata in an audio output format [S1716].

FIG. 18 is a diagram depicting a scenario including image information ofa smart controlling device according to another embodiment of thepresent invention.

Referring to FIG. 18a , if a signal such as “Alice, send a tablet PCimage to a TV” is received from a user, the smart controlling device 110detects both a PC 1810 and a TV 1840.

As shown in FIG. 18b , the smart controlling device 110 transmits acontrol command to the tablet PC 1810 to generate a screen shot imageoutputted on a screen 1820 of the detected tablet PC 1810 and totransmit the screen shot image to the detected TV 1840. In otherembodiments, the tablet PC may transmit currently displayed informationsuch that the tablet PC screen is continuously mirrored on the detectedTV. If necessary, control commands for transmitting the screen shotimage to the detected TV and for controlling playback can be separatelytransmitted to the TV 1840 and the tablet PC. If the screen shot image1830 is received from the tablet PC 1810, the TV 1840 controls anappropriate application to be executed according to the screen shotimage and controls the image to be displayed. Meanwhile, if anapplication was being executed at the TV 1840, the screen shot image1830 can be outputted in various forms including a full screen, a splitscreen, picture-in-picture, and the like.

As discussed, in some embodiments, if a command trigger is received, asmart controlling device 110 can determine whether or not a user ismoving. If movement of the user is detected, data on the movement of theuser is obtained to detect a neighboring display device and determinethe one or more display devices to be selected according to the movementdata. In this case, the data on the movement of the user can include amovement speed of the user, a moving direction of the user, and thelike.

For example, the smart controlling device 110 may select one or moredisplay devices based on the moving speed and the moving direction ofthe user and may be able to control the selected display devices tooutput response data to correspond to the movement of the user afterpower of all of a plurality of the display devices are standby and/orturned on. Accordingly, the response data may be partially displayed ata first display device, and then the remainder may be displayed at asecond display device if the user is detected to be moving from thefirst display device toward the second display device. Alternatively,the response data may be displayed in a directional manner on one ormore of the display devices, such that the response data is displayed tobe sliding in one direction or another according to detected movement ofthe user. In another embodiment, the smart controlling device 110detects a gaze direction of a user using cameras, an eye-trackingsensor, body position sensor, or the like, and may select and control adisplay device with reference to the detected gaze direction of theuser.

According to at least one of the aforementioned embodiments of thepresent invention, it may be able to recognize an input signal andanalyze the recognized input signal based on AI to correspond to theinput signal. Even if the input signal is not a voice input and anoutput signal is not in audio form, the smart controlling device may beable to easily and quickly process the signals. It may be able toprovide an adaptive response using a display of a different neighboringdevice based on environmental factors, user status, location, movementdirection, response data format, and the like. By doing so, it may beable to improve functionality of the smart controlling device as well asfunctionality of the display devices, and further satisfy an intentionof the speaker and increase user convenience.

Meanwhile, the present invention can be implemented with a code readableby a computer in a recording media in which a program is recorded. Therecording media readable by the computer may include all kinds ofrecording devices for storing data capable of being read by the computersystem. The examples of the recording media readable by the computer mayinclude a HDD (hard disk drive), an SSD (solid state disk), a ROM, aRAM, a CD-ROM, a magnetic tape, a floppy disc, an optical data storingdevice and the like. And, the computer may include a controller 180 of aterminal. While the present invention has been described and illustratedherein with reference to the preferred embodiments and diagrams thereof,the present invention may be non-limited to the aforementionedembodiments and it will be apparent to those skilled in the art thatvarious modifications and variations can be made therein withoutdeparting from the spirit and scope of the present specification. Thus,it is intended that the present specification covers the modificationsand variations of this invention that come within the scope of theappended claims and their equivalents.

What is claimed is:
 1. A smart controlling device, comprising: areceiver configured to receive an input comprising a command trigger; acommunication interface configured to transceive data; and a controllerconfigured to: select a display device of one or more known externaldisplay devices; cause a power status of the selected display device tobe changed to a power-on status; and cause a response data correspondingto a first command data received after the command trigger to be outputon a display of the selected display device, wherein the selecteddisplay device is selected from the one or more known external displaydevices based on at least an attribute of the first command data, anattribute of the response data, or an attribute of the selected displaydevice, and wherein the power status of the selected display device ischanged from a power-off status to a stand-by status, and changed fromthe stand-by status to the power-on status.
 2. The device of claim 1,wherein the selected display device is selected based on a function ofthe selected display device being related to the attribute of the firstcommand data or the attribute of the response data.
 3. The device ofclaim 1, wherein: the power status of the selected display device ischanged to the stand-by status after selecting the selected displaydevice; and the power status of the selected display device is changedto the power on status after the received input is obtained for outputon the display of the selected display device.
 4. The device of claim 1,wherein the controller is further configured to transmit a controlcommand to the selected display device to cancel changing the powerstatus of the selected display device when the first command data is notreceived within a threshold period of time following the commandtrigger.
 5. The device of claim 1, wherein the controller is furtherconfigured to obtain second response data corresponding to secondcommand data when the second command data is received within a thresholdperiod of time after the first command data is received or when thesecond command data is received within the threshold period of timeafter the response data corresponding to the first command data isoutput at the selected display device.
 6. The device of claim 1, whereinthe controller is further configured to transmit the response data tothe selected display device via the communication interface fordisplaying the response data via the display of the selected displaydevice.
 7. The device of claim 1, wherein a range for detecting the oneor more known external display devices is expanded based on the firstcommand data.
 8. The device of claim 1, wherein the command triggercorresponds to a voice input, a text input, or a motion gesture.
 9. Thedevice of claim 1, further comprising an output unit, wherein thecontroller is further configured to output an indicator via the outputunit to indicate that the response data is being output at the selecteddisplay device.
 10. A method for controlling a smart controlling device,the method comprising: receiving an input comprising a command trigger;selecting a display device of one or more known external displaydevices; causing a power status of the selected display device to bechanged to a power-on status; and causing a response data correspondingto a first command data received after the command trigger to be outputon a display of the selected display device, wherein the selecteddisplay device is selected from the one or more known external displaydevices based on at least an attribute of the first command data, anattribute of the response data, or an attribute of the selected displaydevice, and wherein the power status of the selected display device ischanged from a power-off status to a stand-by status, and changed fromthe stand-by status to the power-on status.
 11. The method of claim 10,wherein the selected display device is selected based on a function ofthe selected display device being related to the attribute of the firstcommand data or the attribute of the response data.
 12. The method ofclaim 10, wherein: the power status of the selected display device ischanged to the stand-by status after selecting the selected displaydevice; and the power status of the selected display device is changedto the power on status after the received input is obtained for outputon the display of the selected display device.
 13. The method of claim10, further comprising transmitting a control command to the selecteddisplay device to cancel changing the power status of the selecteddisplay device when the first command data is not received within athreshold period of time following the command trigger.
 14. The methodof claim 10, further comprising obtaining second response datacorresponding to second command data when the second command data isreceived within a threshold period of time after the first command datais received or when the second command data is received within thethreshold period of time after the response data corresponding to thefirst command data is output at the selected display device.
 15. Themethod of claim 10, further comprising transmitting the response data tothe selected display device for displaying the response data via thedisplay of the selected display device.
 16. The method of claim 10,wherein a range for detecting the one or more known external displaydevices is expanded based on the first command data.
 17. The method ofclaim 10, wherein the command trigger corresponds to a voice input, atext input, or a motion gesture.
 18. The method of claim 10, furthercomprising outputting an indicator at the smart controlling device toindicate that the response data is being output at the selected displaydevice.